[PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["Fine0830 (via GitHub)" <gi...@apache.org>]

Fine0830 opened a new pull request, #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638998822


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+##### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**

Review Comment:
   As a limit cluster(DB scale can't be very large, such as over 100+ nodes), we don't need to filter nodes. Let's always provide a complete topology.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638990924


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint

Review Comment:
   Time window in `/metrics-windows` is just a query parameter for fetching data from FODC nodes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638964612


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found

Review Comment:
   For 3rd-party, they don't know and care about the IDs. You could use that as internal, but for interface level, only primary IP + port + role + labels matter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2639418676


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,933 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities

Review Comment:
   No authentication: we need simple authN between the sidecar and the proxy. 



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error

Review Comment:
   ```suggestion
   // StreamMetrics(stream MetricsRequest) (stream MetricsResponse) error
   ```
   
   The API linting rules are mandatory: the request message must end with "Request," while the response message must conclude with "Response."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2640120635


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,941 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID            string                    // Unique node identifier
+	NodeRole          databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	PrimaryAddress    Address                  // Primary agent gRPC address
+	SecondaryAddresses map[string]Address      // Secondary addresses with names (e.g., "metrics": Address, "gossip": Address)
+	Labels            map[string]string        // Node labels/metadata
+	RegisteredAt      time.Time                // Registration timestamp
+	LastHeartbeat     time.Time               // Last heartbeat timestamp
+	Status            AgentStatus               // Current agent status
+}
+
+type Address struct {
+	IP   string
+	Port int
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "unconnected"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[AgentKey]*AgentInfo      // Map from agent key (IP+port+role+labels) to agent info
+	mu        sync.RWMutex                 // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+
+type AgentKey struct {
+	IP     string                 // Primary IP address
+	Port   int                    // Primary port
+	Role   databasev1.Role       // Node role
+	Labels map[string]string      // Node labels (used for key matching)
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Creates AgentKey from primary IP + port + role + labels
+- Uses AgentKey as the map key (not nodeID)

Review Comment:
   Please use UUID instead of a key from meaningful fields. I want to avoid duplication of these fields.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638956670


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"

Review Comment:
   @hanahmily I think we could just maintain the active list? Or there are only two statuses, one `online`, the other is `unconnected`, which is only short-term. `unconnected` means the health check failed before reaching the threshold of removing?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638949241


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address

Review Comment:
   We should have a primary address + several secondary addresses with a name.
   And we should separate IP and port via an Address object.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2640335453


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,941 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID            string                    // Unique node identifier
+	NodeRole          databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	PrimaryAddress    Address                  // Primary agent gRPC address
+	SecondaryAddresses map[string]Address      // Secondary addresses with names (e.g., "metrics": Address, "gossip": Address)
+	Labels            map[string]string        // Node labels/metadata
+	RegisteredAt      time.Time                // Registration timestamp
+	LastHeartbeat     time.Time               // Last heartbeat timestamp
+	Status            AgentStatus               // Current agent status
+}
+
+type Address struct {
+	IP   string
+	Port int
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "unconnected"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[AgentKey]*AgentInfo      // Map from agent key (IP+port+role+labels) to agent info
+	mu        sync.RWMutex                 // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+
+type AgentKey struct {
+	IP     string                 // Primary IP address
+	Port   int                    // Primary port
+	Role   databasev1.Role       // Node role
+	Labels map[string]string      // Node labels (used for key matching)
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Creates AgentKey from primary IP + port + role + labels
+- Uses AgentKey as the map key (not nodeID)
+- Validates primary address and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(key AgentKey) error`**
+- Removes agent from registry using AgentKey
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline longer than `--agent-cleanup-timeout` (detected by heartbeat health check)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(key AgentKey) error`**
+- Updates last heartbeat timestamp for agent using AgentKey
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(ip string, port int, role databasev1.Role, labels map[string]string) (*AgentInfo, error)`**
+- Retrieves agent information by primary IP + port + role + labels
+- Creates AgentKey from the provided parameters
+- Looks up agent in registry using AgentKey
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Continuously runs heartbeat health checks regardless of cleanup timeout
+- Unregisters agents that have been offline longer than `--agent-cleanup-timeout`
+- Agents that cannot maintain connection will be removed after the cleanup timeout period
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Minimum**: Must be greater than `--agent-heartbeat-timeout`
+- **Description**: Timeout for automatically unregistering agents that have been offline. Agents that cannot maintain connection will be removed after being offline longer than this timeout. The heartbeat health check continues running regardless of this timeout. This timeout must be greater than `--agent-heartbeat-timeout` to allow for proper health checking.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	logger          *logger.Logger
+}
+
+// Example gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterAgentRequest) (stream RegisterAgentResponse) error
+// StreamMetrics(stream StreamMetricsRequest) (stream StreamMetricsResponse) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	Key         AgentKey              // Agent key (IP+port+role+labels) for registry lookup
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent (includes primary address, role, labels)
+- Creates AgentKey from primary IP + port + role + labels
+- Validates registration information
+- Registers agent with AgentRegistry using AgentKey
+- Sends registration responses
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Sends metrics requests from Proxy to agent (on-demand collection)
+- Receives metrics from agent at Proxy in response to requests
+- Proxy initiates by sending a metrics request via StreamMetricsResponse when an external client queries metrics
+- Agent responds with StreamMetricsRequest containing the collected metrics
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Extract agent key (primary IP + port + role + labels) from the connection
+  3. Check if this is the last active stream for the agent
+  4. If all streams are closed, call `AgentRegistry.UnregisterAgent(agentKey)` to clean up
+  5. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration occurs after being offline longer than `--agent-cleanup-timeout`. Heartbeat health checks continue running regardless.
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Time Window Filtering**: Filters metrics by time window when requested by external clients (agents filter based on Flight Recorder data)
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent with time window filter (if specified)
+- Agents retrieve metrics from Flight Recorder (in-memory storage) filtered by time window and respond
+- Waits for StreamMetricsRequest responses from agents (with timeout)
+- Identifies agent connection from stream context and looks up AgentKey
+- Enriches metrics with node metadata (IP, port, role, labels) from AgentKey
+- Combines metrics from all agents into a single list
+- Returns aggregated metrics (not stored, only returned)
+- Returns error if collection fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with no time filter to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint without time parameters
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with time window filter
+- Agents filter metrics from Flight Recorder by the specified time range
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+##### Configuration Flags
+
+*Note: No configuration flags needed for MetricsAggregator since metrics are collected on-demand and not stored.*
+
+#### 1.4 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns latest metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional) - filters metrics by start time (agents filter from Flight Recorder)
+  - `end_time`: End time for time window (optional) - filters metrics by end time (agents filter from Flight Recorder)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Node relationships
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+##### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+  nodeIP         string
+  nodePort       int

Review Comment:
   These two fields are not formatted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["codecov-commenter (via GitHub)" <gi...@apache.org>]

codecov-commenter commented on PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#issuecomment-3680857702

   ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/911?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 29.07%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c8de027`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/c8de0277e337b8b3ecb718c767bf263074daafb5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 82 commits behind head on main.
   > :exclamation:  There is a different number of reports uploaded between BASE (3530dd9) and HEAD (c8de027). Click for more details.
   > 
   > <details><summary>HEAD has 1 upload less than BASE</summary>
   >
   >| Flag | BASE (3530dd9) | HEAD (c8de027) |
   >|------|------|------|
   >||1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##             main     #911       +/-   ##
   ===========================================
   - Coverage   45.97%   29.07%   -16.91%     
   ===========================================
     Files         328      118      -210     
     Lines       55505    12183    -43322     
   ===========================================
   - Hits        25520     3542    -21978     
   + Misses      27909     8302    -19607     
   + Partials     2076      339     -1737     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/911/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/911/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `29.07% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/911?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2639006250


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+##### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {

Review Comment:
   Can't find upstream-downstream nodes relationships in the topology. Where are they?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638994161


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context

Review Comment:
   By that, one day, when we need FODC proxy in the cluster mode, we could allow FODC agent to register to multiple proxies.
   This kind of light-weight architecture is more helpful in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2639030618


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+##### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems

Review Comment:
   This should respond to all the latest metric values from all nodes. You could add proxy's own metrics into it as well.
   But other nodes' metrics are critical.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638950145


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address

Review Comment:
   e.g.
   
   - gossip address: ip+port



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["Copilot (via GitHub)" <gi...@apache.org>]

Copilot commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638914359


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails

Review Comment:
   The error message description states "Returns error if agent not found" but doesn't document what happens when metrics collection from an agent fails during the request. The function signature shows it returns a map and an error, but it's unclear if partial results are returned when some agents succeed and others fail, or if the entire operation fails. This error handling behavior should be documented.



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics

Review Comment:
   The comment incorrectly states the bi-directional stream direction. The protobuf service definition shows `rpc StreamMetrics(stream MetricsMessage) returns (stream MetricsRequest)`, which means the Agent sends MetricsMessage and receives MetricsRequest from the Proxy. However, the comment states "Receives metrics requests from Proxy (on-demand collection)" followed by "Sends metrics from agent to Proxy when requested", which reverses the correct order. The Proxy sends MetricsRequest first, and the Agent responds with MetricsMessage.
   ```suggestion
   - Sends metrics requests from Proxy to agent (on-demand collection)
   - Receives metrics from agent at Proxy in response to requests
   - Proxy initiates by sending a metrics request via MetricsRequest when an external client queries metrics
   - Agent responds with MetricsMessage containing the collected metrics
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)

Review Comment:
   The gRPC service definition comment is misleading. It states "gRPC service methods (to be defined in proto)" but then shows the actual method signatures. Either the methods are already defined (in which case remove the "to be defined" comment), or they haven't been defined yet (in which case the signatures shown are just proposals). This ambiguity should be clarified.
   ```suggestion
   // Example gRPC service methods (to be defined in proto)
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+

Review Comment:
   The comment describing the bi-directional stream for configuration has the same issue as the metrics stream. According to the protobuf definition `rpc StreamNodesConfigurations(stream ConfigurationMessage) returns (stream ConfigurationRequest)`, the Proxy sends ConfigurationRequest and receives ConfigurationMessage from the Agent. The description order should reflect that the Proxy sends a request first via ConfigurationRequest, then the Agent responds with ConfigurationMessage.
   ```suggestion
   - Handles bi-directional configuration streaming between Proxy and Agent
   - Proxy sends configuration requests to Agent via `ConfigurationRequest` (on-demand collection when external client queries configurations)
   - Agent responds with `ConfigurationMessage` containing collected configuration, sending configuration data back to the Proxy
   - Ensures configuration responses are correlated to the corresponding requests
   - Forwards configuration to ConfigurationCollector for storage
   - Manages stream lifecycle
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+#### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	topologyManager     *TopologyManager
+	configCollector     *ConfigurationCollector
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+	nodeID         string
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	sessionID      string
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterRequest with node ID, role, and labels
+- Receives RegisterResponse with session ID and heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in MetricsMessage
+- Sends metrics to Proxy via StreamMetrics() stream
+- Includes node ID, session ID, and timestamp
+- Returns error if retrieval or send fails
+
+**`StartConfigurationStream(ctx context.Context) error`**
+- Establishes bi-directional configuration stream with Proxy
+- Maintains stream for on-demand configuration requests
+- Listens for configuration request commands from Proxy
+- When request received, collects configuration from BanyanDB process and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process when requested by Proxy
+- Finds BanyanDB process (by PID or process name)
+- Reads `/proc/<pid>/cmdline` for command-line flags
+- Reads `/proc/<pid>/environ` for environment variables
+- Sanitizes sensitive data
+- Packages configuration in ConfigurationMessage
+- Sends configuration to Proxy via StreamNodesConfigurations() stream
+- Includes node ID, session ID, and collected timestamp
+- Returns error if collection or send fails
+
+**`SendHeartbeat(ctx context.Context) error`**
+- Sends heartbeat to Proxy
+- Updates connection status
+- Returns error if heartbeat fails
+
+**`Disconnect() error`**
+- Closes connection to Proxy
+- Cleans up resources
+- Returns error if disconnect fails
+
+##### Configuration Flags
+
+**`--proxy-addr`**
+- **Type**: `string`
+- **Default**: `localhost:17900`
+- **Description**: FODC Proxy gRPC address
+
+**`--node-id`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Unique identifier for this BanyanDB node. Must be unique across the cluster. Typically matches the node's identity in the BanyanDB cluster.
+
+**`--node-role`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Role of this BanyanDB node. Valid values: `liaison`, `datanode-hot`, `datanode-warm`, `datanode-cold`, etc. Must match the node's actual role in the cluster.
+
+**`--node-labels`**
+- **Type**: `string` (comma-separated key=value pairs)
+- **Default**: (optional)
+- **Description**: Labels/metadata for this node. Format: `key1=value1,key2=value2`. Examples: `zone=us-west-1,env=production`. Used for filtering and grouping nodes in the Proxy.
+
+**`--heartbeat-interval`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: Interval for sending heartbeats to Proxy. Note: The Proxy may override this value in RegisterResponse.
+
+**`--reconnect-interval`**
+- **Type**: `duration`
+- **Default**: `5s`
+- **Description**: Interval for reconnection attempts when connection to Proxy is lost
+
+#### 2.2 Configuration Collector Component
+
+**Purpose**: Collects configuration from the BanyanDB process running on the same node
+
+##### Core Responsibilities
+
+- **Process Discovery**: Finds the BanyanDB process by PID or process name
+- **Command-Line Collection**: Reads command-line flags from `/proc/<pid>/cmdline`
+- **Environment Collection**: Reads environment variables from `/proc/<pid>/environ`
+- **Data Sanitization**: Sanitizes sensitive data (passwords, tokens, etc.)
+- **Configuration Packaging**: Packages configuration data for transmission to Proxy
+
+##### Core Types
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	banyandbPID    int                    // BanyanDB process ID
+	banyandbProcName string               // BanyanDB process name
+	proxyClient    *ProxyClient
+	logger         *logger.Logger
+}
+```
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID        string
+	NodeRole      databasev1.Role
+	StartupParams map[string]string    // Command-line flags from /proc/<pid>/cmdline
+	EnvVars       map[string]string    // Environment variables from /proc/<pid>/environ (sanitized)
+	RuntimeConfig map[string]string    // Runtime configurations (if available)
+	CollectedAt   time.Time
+}
+```
+
+##### Key Functions
+
+**`FindBanyanDBProcess() (int, error)`**
+- Finds BanyanDB process by PID (if known) or process name
+- Searches `/proc` filesystem for matching process
+- Returns process ID or error if not found
+
+**`CollectStartupParams(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/cmdline` file
+- Parses null-separated command-line arguments
+- Extracts flags and values into map
+- Returns error if read fails
+
+**`CollectEnvVars(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/environ` file
+- Parses null-separated environment variables
+- Sanitizes sensitive data (passwords, tokens, API keys, etc.)
+- Returns sanitized environment variables map
+- Returns error if read fails
+
+**`CollectConfiguration() (*NodeConfiguration, error)`**
+- Finds BanyanDB process
+- Collects startup parameters from `/proc/<pid>/cmdline`
+- Collects environment variables from `/proc/<pid>/environ`
+- Packages configuration into NodeConfiguration struct
+- Returns error if collection fails
+
+**`SendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process
+- Sends configuration to Proxy via gRPC
+- Returns error if send fails
+
+##### Configuration Flags
+
+**`--banyandb-pid`**
+- **Type**: `int`
+- **Default**: (optional, auto-detected if not specified)
+- **Description**: BanyanDB process ID. If not specified, agent will search for process by name.
+
+**`--banyandb-proc-name`**
+- **Type**: `string`
+- **Default**: `banyandb`
+- **Description**: BanyanDB process name to search for if PID is not provided
+
+**`--config-collection-interval`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Interval for collecting and sending configuration updates to Proxy
+
+## API Design
+
+### gRPC API
+
+#### Service Definition
+
+```protobuf
+syntax = "proto3";
+
+package banyandb.fodc.v1;
+
+import "google/protobuf/timestamp.proto";
+
+service FODCService {
+  // Bi-directional stream for agent registration and heartbeat
+  rpc RegisterAgent(stream RegisterRequest) returns (stream RegisterResponse);
+  
+  // Bi-directional stream for metrics
+  rpc StreamMetrics(stream MetricsMessage) returns (stream MetricsRequest);
+  
+  // Bi-directional stream for node configurations
+  rpc StreamNodesConfigurations(stream ConfigurationMessage) returns (stream ConfigurationRequest);
+}
+
+message RegisterRequest {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> labels = 3;
+}
+
+message RegisterResponse {
+  string session_id = 1;
+  bool success = 2;
+  string message = 3;
+  int64 heartbeat_interval_seconds = 4;
+}
+
+message MetricsMessage {
+  string node_id = 1;
+  string session_id = 2;
+  repeated Metric metrics = 3;
+  google.protobuf.Timestamp timestamp = 4;
+}
+
+message Metric {
+  string name = 1;
+  map<string, string> labels = 2;
+  double value = 3;
+  string description = 4;
+}
+
+message MetricsRequest {
+  google.protobuf.Timestamp start_time = 1;  // Optional start time for time window
+  google.protobuf.Timestamp end_time = 2;    // Optional end time for time window
+}
+
+message ConfigurationMessage {
+  string node_id = 1;
+  string session_id = 2;
+  NodeConfiguration configuration = 3;
+  google.protobuf.Timestamp collected_at = 4;
+}
+
+message NodeConfiguration {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> startup_params = 3;  // Command-line flags from /proc/<pid>/cmdline

Review Comment:
   `NodeConfiguration.startup_params` is defined to include all command-line flags from `/proc/<pid>/cmdline`, but there is no mention of sanitizing or filtering out sensitive flags. BanyanDB already accepts secrets such as etcd credentials via CLI flags (for example `--etcd-password`), so including raw `startup_params` here means those passwords will be collected and propagated to the Proxy in cleartext. You should either avoid collecting secret-bearing flags or aggressively redact/whitelist-safe keys before they are stored or exposed to other components.
   ```suggestion
     map<string, string> startup_params = 3;  // Sanitized command-line flags from /proc/<pid>/cmdline (sensitive flags redacted/omitted)
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails

Review Comment:
   Similar to the metrics collection function, this function's error handling behavior is not fully documented. It's unclear whether partial results are returned when some agents fail to respond, or if a single agent failure causes the entire operation to fail. Given that this affects the `/cluster/config` endpoint behavior, the error handling strategy should be clearly documented.
   ```suggestion
   - Stores configurations per node that responded successfully
   - Returns a map from node ID to configuration; the map may be partial if some agents fail or time out
   - Returns a non-nil error if any agent collection fails, the context is canceled, or no configurations can be collected; callers should be prepared to handle a non-nil result map together with a non-nil error
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+#### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	topologyManager     *TopologyManager
+	configCollector     *ConfigurationCollector
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+	nodeID         string
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	sessionID      string
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterRequest with node ID, role, and labels
+- Receives RegisterResponse with session ID and heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in MetricsMessage
+- Sends metrics to Proxy via StreamMetrics() stream
+- Includes node ID, session ID, and timestamp
+- Returns error if retrieval or send fails
+
+**`StartConfigurationStream(ctx context.Context) error`**
+- Establishes bi-directional configuration stream with Proxy
+- Maintains stream for on-demand configuration requests
+- Listens for configuration request commands from Proxy
+- When request received, collects configuration from BanyanDB process and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process when requested by Proxy
+- Finds BanyanDB process (by PID or process name)
+- Reads `/proc/<pid>/cmdline` for command-line flags
+- Reads `/proc/<pid>/environ` for environment variables
+- Sanitizes sensitive data
+- Packages configuration in ConfigurationMessage
+- Sends configuration to Proxy via StreamNodesConfigurations() stream
+- Includes node ID, session ID, and collected timestamp
+- Returns error if collection or send fails
+
+**`SendHeartbeat(ctx context.Context) error`**
+- Sends heartbeat to Proxy
+- Updates connection status
+- Returns error if heartbeat fails
+
+**`Disconnect() error`**
+- Closes connection to Proxy
+- Cleans up resources
+- Returns error if disconnect fails
+
+##### Configuration Flags
+
+**`--proxy-addr`**
+- **Type**: `string`
+- **Default**: `localhost:17900`
+- **Description**: FODC Proxy gRPC address
+
+**`--node-id`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Unique identifier for this BanyanDB node. Must be unique across the cluster. Typically matches the node's identity in the BanyanDB cluster.
+
+**`--node-role`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Role of this BanyanDB node. Valid values: `liaison`, `datanode-hot`, `datanode-warm`, `datanode-cold`, etc. Must match the node's actual role in the cluster.
+
+**`--node-labels`**
+- **Type**: `string` (comma-separated key=value pairs)
+- **Default**: (optional)
+- **Description**: Labels/metadata for this node. Format: `key1=value1,key2=value2`. Examples: `zone=us-west-1,env=production`. Used for filtering and grouping nodes in the Proxy.
+
+**`--heartbeat-interval`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: Interval for sending heartbeats to Proxy. Note: The Proxy may override this value in RegisterResponse.
+
+**`--reconnect-interval`**
+- **Type**: `duration`
+- **Default**: `5s`
+- **Description**: Interval for reconnection attempts when connection to Proxy is lost
+
+#### 2.2 Configuration Collector Component
+
+**Purpose**: Collects configuration from the BanyanDB process running on the same node
+
+##### Core Responsibilities
+
+- **Process Discovery**: Finds the BanyanDB process by PID or process name
+- **Command-Line Collection**: Reads command-line flags from `/proc/<pid>/cmdline`
+- **Environment Collection**: Reads environment variables from `/proc/<pid>/environ`
+- **Data Sanitization**: Sanitizes sensitive data (passwords, tokens, etc.)
+- **Configuration Packaging**: Packages configuration data for transmission to Proxy
+
+##### Core Types
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	banyandbPID    int                    // BanyanDB process ID
+	banyandbProcName string               // BanyanDB process name
+	proxyClient    *ProxyClient
+	logger         *logger.Logger
+}
+```
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID        string
+	NodeRole      databasev1.Role
+	StartupParams map[string]string    // Command-line flags from /proc/<pid>/cmdline
+	EnvVars       map[string]string    // Environment variables from /proc/<pid>/environ (sanitized)
+	RuntimeConfig map[string]string    // Runtime configurations (if available)
+	CollectedAt   time.Time
+}
+```
+
+##### Key Functions
+
+**`FindBanyanDBProcess() (int, error)`**
+- Finds BanyanDB process by PID (if known) or process name
+- Searches `/proc` filesystem for matching process
+- Returns process ID or error if not found
+
+**`CollectStartupParams(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/cmdline` file
+- Parses null-separated command-line arguments
+- Extracts flags and values into map
+- Returns error if read fails
+
+**`CollectEnvVars(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/environ` file
+- Parses null-separated environment variables
+- Sanitizes sensitive data (passwords, tokens, API keys, etc.)
+- Returns sanitized environment variables map
+- Returns error if read fails
+
+**`CollectConfiguration() (*NodeConfiguration, error)`**
+- Finds BanyanDB process
+- Collects startup parameters from `/proc/<pid>/cmdline`
+- Collects environment variables from `/proc/<pid>/environ`
+- Packages configuration into NodeConfiguration struct
+- Returns error if collection fails
+
+**`SendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process
+- Sends configuration to Proxy via gRPC
+- Returns error if send fails
+
+##### Configuration Flags
+
+**`--banyandb-pid`**
+- **Type**: `int`
+- **Default**: (optional, auto-detected if not specified)
+- **Description**: BanyanDB process ID. If not specified, agent will search for process by name.
+
+**`--banyandb-proc-name`**
+- **Type**: `string`
+- **Default**: `banyandb`
+- **Description**: BanyanDB process name to search for if PID is not provided
+
+**`--config-collection-interval`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Interval for collecting and sending configuration updates to Proxy
+
+## API Design
+
+### gRPC API
+
+#### Service Definition
+
+```protobuf
+syntax = "proto3";
+
+package banyandb.fodc.v1;
+
+import "google/protobuf/timestamp.proto";
+
+service FODCService {
+  // Bi-directional stream for agent registration and heartbeat
+  rpc RegisterAgent(stream RegisterRequest) returns (stream RegisterResponse);
+  
+  // Bi-directional stream for metrics
+  rpc StreamMetrics(stream MetricsMessage) returns (stream MetricsRequest);
+  
+  // Bi-directional stream for node configurations
+  rpc StreamNodesConfigurations(stream ConfigurationMessage) returns (stream ConfigurationRequest);
+}
+
+message RegisterRequest {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> labels = 3;
+}
+
+message RegisterResponse {
+  string session_id = 1;
+  bool success = 2;
+  string message = 3;
+  int64 heartbeat_interval_seconds = 4;
+}
+
+message MetricsMessage {
+  string node_id = 1;
+  string session_id = 2;
+  repeated Metric metrics = 3;
+  google.protobuf.Timestamp timestamp = 4;
+}
+
+message Metric {
+  string name = 1;
+  map<string, string> labels = 2;
+  double value = 3;
+  string description = 4;
+}
+
+message MetricsRequest {
+  google.protobuf.Timestamp start_time = 1;  // Optional start time for time window
+  google.protobuf.Timestamp end_time = 2;    // Optional end time for time window
+}
+
+message ConfigurationMessage {
+  string node_id = 1;
+  string session_id = 2;
+  NodeConfiguration configuration = 3;
+  google.protobuf.Timestamp collected_at = 4;
+}
+
+message NodeConfiguration {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> startup_params = 3;  // Command-line flags from /proc/<pid>/cmdline
+  map<string, string> env_vars = 4;         // Environment variables from /proc/<pid>/environ (sanitized)
+  map<string, string> runtime_config = 5;   // Runtime configurations (if available)
+}
+
+message ConfigurationRequest {
+  bool request_all = 1;  // Request configurations from all agents
+  repeated string node_ids = 2;  // Specific node IDs to request (if request_all is false)
+}
+```
+
+### REST API
+
+#### Endpoints
+
+**`GET /metrics`**
+- **Description**: Returns proxy's own metrics in Prometheus format
+- **Response**: Prometheus text format
+- **Example**:
+```
+# HELP fodc_proxy_agents_total Total number of registered agents
+# TYPE fodc_proxy_agents_total gauge
+fodc_proxy_agents_total 5
+
+# HELP fodc_proxy_grpc_requests_total Total gRPC requests
+# TYPE fodc_proxy_grpc_requests_total counter
+fodc_proxy_grpc_requests_total 1234
+```
+
+**`GET /metrics-windows`**
+- **Description**: Returns metrics within time window from all agents
+- **Query Parameters**:
+  - `start_time` (optional): RFC3339 timestamp
+  - `end_time` (optional): RFC3339 timestamp
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- **Response**: Prometheus text format with node metadata labels
+- **Example**:
+```
+# HELP banyandb_stream_tst_inverted_index_total_doc_count Total document count
+# TYPE banyandb_stream_tst_inverted_index_total_doc_count gauge
+banyandb_stream_tst_inverted_index_total_doc_count{index="test",node_id="node1",node_role="datanode-hot"} 12345
+```
+
+**`GET /cluster`**
+- **Description**: Returns cluster topology and node information
+- **Query Parameters**:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role
+- **Response**: JSON
+- **Example**:
+```json
+{
+  "nodes": [
+    {
+      "node_id": "node1",
+      "node_role": "liaison",
+      "address": "192.168.1.1:17900",
+      "labels": {
+        "zone": "us-west-1"
+      },
+      "status": "online",
+      "last_heartbeat": "2024-01-01T12:00:00Z",
+      "runtime_metrics": {
+        "load": 0.5,
+        "health": "healthy"
+      }
+    }
+  ],
+  "updated_at": "2024-01-01T12:00:00Z"
+}
+```
+
+**`GET /cluster/config`**
+- **Description**: Returns node configurations
+- **Query Parameters**:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role
+- **Response**: JSON
+- **Example**:
+```json
+{
+  "configurations": [
+    {
+      "node_id": "node1",
+      "node_role": "liaison",
+      "startup_params": {
+        "--listen-addr": ":17900",
+        "--data-path": "/data"
+      },
+      "runtime_config": {
+        "max_connections": 1000
+      },
+      "collected_at": "2024-01-01T12:00:00Z"
+    }
+  ]
+}

Review Comment:
   The `/cluster/config` endpoint is documented to return node configurations including `startup_params`, which in this design contain raw command-line flags from BanyanDB processes. Because those flags can legitimately contain secrets (for example `--etcd-password` from the installation guide), any client that can reach this HTTP endpoint will be able to retrieve infrastructure credentials in cleartext. Restrict what this API returns by default (e.g., omit or redact secret-bearing fields) or gate it behind strong authentication and encryption so configuration inspection does not leak sensitive data.



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations

Review Comment:
   The field type for RuntimeConfig is inconsistent. In line 420, it's defined as `map[string]interface{}`, but in line 741 (Agent's NodeConfiguration), it's defined as `map[string]string`. These should be consistent across Proxy and Agent components since they represent the same data structure being transmitted via gRPC.
   ```suggestion
   	RuntimeConfig    map[string]string    // Runtime configurations
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+#### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	topologyManager     *TopologyManager
+	configCollector     *ConfigurationCollector
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+	nodeID         string
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	sessionID      string
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterRequest with node ID, role, and labels
+- Receives RegisterResponse with session ID and heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in MetricsMessage
+- Sends metrics to Proxy via StreamMetrics() stream
+- Includes node ID, session ID, and timestamp
+- Returns error if retrieval or send fails
+
+**`StartConfigurationStream(ctx context.Context) error`**
+- Establishes bi-directional configuration stream with Proxy
+- Maintains stream for on-demand configuration requests
+- Listens for configuration request commands from Proxy
+- When request received, collects configuration from BanyanDB process and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process when requested by Proxy
+- Finds BanyanDB process (by PID or process name)
+- Reads `/proc/<pid>/cmdline` for command-line flags
+- Reads `/proc/<pid>/environ` for environment variables
+- Sanitizes sensitive data
+- Packages configuration in ConfigurationMessage
+- Sends configuration to Proxy via StreamNodesConfigurations() stream
+- Includes node ID, session ID, and collected timestamp
+- Returns error if collection or send fails
+
+**`SendHeartbeat(ctx context.Context) error`**
+- Sends heartbeat to Proxy
+- Updates connection status
+- Returns error if heartbeat fails
+
+**`Disconnect() error`**
+- Closes connection to Proxy
+- Cleans up resources
+- Returns error if disconnect fails
+
+##### Configuration Flags
+
+**`--proxy-addr`**
+- **Type**: `string`
+- **Default**: `localhost:17900`
+- **Description**: FODC Proxy gRPC address
+
+**`--node-id`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Unique identifier for this BanyanDB node. Must be unique across the cluster. Typically matches the node's identity in the BanyanDB cluster.
+
+**`--node-role`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Role of this BanyanDB node. Valid values: `liaison`, `datanode-hot`, `datanode-warm`, `datanode-cold`, etc. Must match the node's actual role in the cluster.
+
+**`--node-labels`**
+- **Type**: `string` (comma-separated key=value pairs)
+- **Default**: (optional)
+- **Description**: Labels/metadata for this node. Format: `key1=value1,key2=value2`. Examples: `zone=us-west-1,env=production`. Used for filtering and grouping nodes in the Proxy.
+
+**`--heartbeat-interval`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: Interval for sending heartbeats to Proxy. Note: The Proxy may override this value in RegisterResponse.
+
+**`--reconnect-interval`**
+- **Type**: `duration`
+- **Default**: `5s`
+- **Description**: Interval for reconnection attempts when connection to Proxy is lost
+
+#### 2.2 Configuration Collector Component
+
+**Purpose**: Collects configuration from the BanyanDB process running on the same node
+
+##### Core Responsibilities
+
+- **Process Discovery**: Finds the BanyanDB process by PID or process name
+- **Command-Line Collection**: Reads command-line flags from `/proc/<pid>/cmdline`
+- **Environment Collection**: Reads environment variables from `/proc/<pid>/environ`
+- **Data Sanitization**: Sanitizes sensitive data (passwords, tokens, etc.)
+- **Configuration Packaging**: Packages configuration data for transmission to Proxy
+
+##### Core Types
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	banyandbPID    int                    // BanyanDB process ID
+	banyandbProcName string               // BanyanDB process name
+	proxyClient    *ProxyClient
+	logger         *logger.Logger
+}
+```
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID        string
+	NodeRole      databasev1.Role
+	StartupParams map[string]string    // Command-line flags from /proc/<pid>/cmdline
+	EnvVars       map[string]string    // Environment variables from /proc/<pid>/environ (sanitized)
+	RuntimeConfig map[string]string    // Runtime configurations (if available)
+	CollectedAt   time.Time
+}
+```
+
+##### Key Functions
+
+**`FindBanyanDBProcess() (int, error)`**
+- Finds BanyanDB process by PID (if known) or process name
+- Searches `/proc` filesystem for matching process
+- Returns process ID or error if not found
+
+**`CollectStartupParams(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/cmdline` file
+- Parses null-separated command-line arguments
+- Extracts flags and values into map
+- Returns error if read fails
+
+**`CollectEnvVars(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/environ` file
+- Parses null-separated environment variables
+- Sanitizes sensitive data (passwords, tokens, API keys, etc.)
+- Returns sanitized environment variables map
+- Returns error if read fails
+
+**`CollectConfiguration() (*NodeConfiguration, error)`**
+- Finds BanyanDB process
+- Collects startup parameters from `/proc/<pid>/cmdline`
+- Collects environment variables from `/proc/<pid>/environ`
+- Packages configuration into NodeConfiguration struct
+- Returns error if collection fails
+
+**`SendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process
+- Sends configuration to Proxy via gRPC
+- Returns error if send fails
+
+##### Configuration Flags
+
+**`--banyandb-pid`**
+- **Type**: `int`
+- **Default**: (optional, auto-detected if not specified)
+- **Description**: BanyanDB process ID. If not specified, agent will search for process by name.
+
+**`--banyandb-proc-name`**
+- **Type**: `string`
+- **Default**: `banyandb`
+- **Description**: BanyanDB process name to search for if PID is not provided
+
+**`--config-collection-interval`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Interval for collecting and sending configuration updates to Proxy

Review Comment:
   The configuration flag description is incomplete and potentially misleading. While it mentions that configurations are collected "on-demand" when requested, the flag description suggests periodic collection at a 5-minute interval. This contradicts the on-demand collection model described throughout the document (lines 442-449, 1074-1077). Either the flag should be removed/repurposed, or the collection model should be clarified to explain if there's both periodic and on-demand collection.
   ```suggestion
   - **Description**: Interval for the agent’s background periodic configuration collection and sending of configuration updates to the Proxy (used in addition to on-demand collection requested by the Proxy)
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+#### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	topologyManager     *TopologyManager
+	configCollector     *ConfigurationCollector
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+	nodeID         string
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	sessionID      string
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterRequest with node ID, role, and labels
+- Receives RegisterResponse with session ID and heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in MetricsMessage
+- Sends metrics to Proxy via StreamMetrics() stream
+- Includes node ID, session ID, and timestamp
+- Returns error if retrieval or send fails
+
+**`StartConfigurationStream(ctx context.Context) error`**
+- Establishes bi-directional configuration stream with Proxy
+- Maintains stream for on-demand configuration requests
+- Listens for configuration request commands from Proxy
+- When request received, collects configuration from BanyanDB process and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process when requested by Proxy
+- Finds BanyanDB process (by PID or process name)
+- Reads `/proc/<pid>/cmdline` for command-line flags
+- Reads `/proc/<pid>/environ` for environment variables
+- Sanitizes sensitive data
+- Packages configuration in ConfigurationMessage
+- Sends configuration to Proxy via StreamNodesConfigurations() stream
+- Includes node ID, session ID, and collected timestamp
+- Returns error if collection or send fails

Review Comment:
   The description states "Reads configuration from BanyanDB process", but the actual implementation details in lines 1084-1089 show it reads from `/proc/<pid>/cmdline` and `/proc/<pid>/environ`, which is Linux-specific. This design limitation should be explicitly documented here to clarify that this approach only works on Linux systems and may require different implementation for other operating systems.
   ```suggestion
   - When request received, collects configuration from the BanyanDB process (on Linux systems via `/proc`) and sends response
   - Returns error if stream fails
   
   **`RetrieveAndSendConfiguration(ctx context.Context) error`**
   - Collects configuration from BanyanDB process when requested by Proxy
   - Finds BanyanDB process (by PID or process name)
   - On Linux systems, reads `/proc/<pid>/cmdline` for command-line flags
   - On Linux systems, reads `/proc/<pid>/environ` for environment variables
   - Sanitizes sensitive data
   - Packages configuration in ConfigurationMessage
   - Sends configuration to Proxy via StreamNodesConfigurations() stream
   - Includes node ID, session ID, and collected timestamp
   - Returns error if collection or send fails
   - Note: The `/proc`-based configuration collection is Linux-specific; other operating systems require alternative mechanisms to retrieve equivalent process configuration.
   ```



##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+#### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+#### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+#### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+#### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+#### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+#### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+#### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+#### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context
+
+#### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	metricsWindow *MetricsWindow      // Sliding window storage
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsWindow struct {
+	metrics    map[string][]*AggregatedMetric  // Map from metric key to time series
+	windowSize time.Duration                   // Time window size
+	mu         sync.RWMutex
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+#### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) (map[string][]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent
+- Agents retrieve metrics from Flight Recorder (in-memory storage) and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Returns map from node ID to aggregated metrics
+- Returns error if collection fails
+
+**`AggregateMetrics(nodeID string, rawMetrics []*RawMetric) error`**
+- Processes raw metrics received from an agent
+- Enriches metrics with node metadata from registry
+- Stores metrics in time window
+- Returns error if aggregation fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Filters metrics by specified time window
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+**`GetMetricsByNode(nodeID string) []*AggregatedMetric`**
+- Returns metrics for specific node
+- Useful for node-specific queries
+
+**`GetMetricsByRole(role databasev1.Role) []*AggregatedMetric`**
+- Returns metrics filtered by node role
+- Useful for role-specific analysis
+
+**`CleanupOldMetrics(beforeTime time.Time) error`**
+- Removes metrics older than specified time
+- Prevents memory growth
+- Called periodically
+
+#### Configuration Flags
+
+**`--metrics-window-size`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Time window size for metrics storage
+
+**`--metrics-cleanup-interval`**
+- **Type**: `duration`
+- **Default**: `1m`
+- **Description**: Interval for cleaning up old metrics
+
+#### 1.4 Topology Manager Component
+
+**Purpose**: Maintains cluster topology view based on agent registrations
+
+##### Core Responsibilities
+
+- **Topology Building**: Constructs cluster hierarchy from agent registrations
+- **Role Management**: Tracks nodes by role (liaison, datanode-hot, etc.)
+- **Status Tracking**: Monitors node status and health
+- **Topology Queries**: Provides topology information via API
+
+##### Core Types
+
+**`ClusterTopology`**
+```go
+type ClusterTopology struct {
+	Nodes      []*NodeInfo              // All nodes in cluster
+	ByRole     map[databasev1.Role][]*NodeInfo  // Nodes grouped by role
+	UpdatedAt  time.Time                // Last update timestamp
+	mu         sync.RWMutex
+}
+
+type NodeInfo struct {
+	NodeID      string
+	NodeRole    databasev1.Role
+	Address     string
+	Labels      map[string]string
+	Status      AgentStatus
+	LastHeartbeat time.Time
+	RuntimeMetrics *RuntimeMetrics  // Optional runtime metrics
+}
+
+type RuntimeMetrics struct {
+	Load    float64  // Optional load indicator
+	Health  string   // Optional health status
+}
+```
+
+**`TopologyManager`**
+```go
+type TopologyManager struct {
+	registry  *AgentRegistry
+	topology  *ClusterTopology
+	logger    *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`UpdateTopology() error`**
+- Rebuilds topology from agent registry
+- Groups nodes by role
+- Updates topology timestamp
+- Called when agents register/unregister
+
+**`GetTopology() *ClusterTopology`**
+- Returns current cluster topology
+- Thread-safe read operation
+
+**`GetNodesByRole(role databasev1.Role) []*NodeInfo`**
+- Returns nodes filtered by role
+- Useful for role-specific operations
+
+**`GetNode(nodeID string) (*NodeInfo, error)`**
+- Returns node information by ID
+- Returns error if node not found
+
+#### 1.5 Configuration Collector Component
+
+**Purpose**: Collects and aggregates node configurations from agents
+
+##### Core Responsibilities
+
+- **Configuration Collection**: Receives configuration data from agents
+- **Storage**: Stores configurations per node
+- **Consistency Checking**: Provides utilities for comparing configurations
+- **Configuration Exposure**: Exposes configurations via REST API
+
+##### Core Types
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID           string
+	NodeRole         databasev1.Role
+	StartupParams    map[string]string    // Command-line flags
+	EnvVars          map[string]string    // Environment variables (sanitized)
+	RuntimeConfig    map[string]interface{} // Runtime configurations
+	CollectedAt      time.Time
+}
+```
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	registry  *AgentRegistry
+	configs   map[string]*NodeConfiguration  // Map from node ID to config
+	mu        sync.RWMutex
+	logger    *logger.Logger
+}
+
+type ConfigurationFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectConfigurationsFromAgents(ctx context.Context, filter *ConfigurationFilter) (map[string]*NodeConfiguration, error)`**
+- Requests configurations from all agents (or filtered agents) when external client queries
+- Sends configuration request via StreamNodesConfigurations() to each agent
+- Agents collect configuration from BanyanDB process and respond
+- Waits for ConfigurationMessage responses from agents (with timeout)
+- Stores configurations per node
+- Returns map from node ID to configuration
+- Returns error if collection fails
+
+**`CollectConfiguration(nodeID string, config *NodeConfiguration) error`**
+- Stores configuration for a node
+- Updates existing configuration if present
+- Returns error if collection fails
+
+**`GetConfiguration(nodeID string) (*NodeConfiguration, error)`**
+- Retrieves configuration for specific node
+- Triggers on-demand collection from agent if not cached or stale
+- Returns error if not found
+
+**`GetAllConfigurations(ctx context.Context) (map[string]*NodeConfiguration, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectConfigurationsFromAgents() to get current configurations
+- Returns all node configurations
+- Used for `/cluster/config` endpoint
+- Returns error if collection fails
+
+**`GetConfigurationsByRole(ctx context.Context, role databasev1.Role) ([]*NodeConfiguration, error)`**
+- Triggers on-demand collection from agents filtered by role
+- Returns configurations filtered by role
+- Useful for role-based consistency checks
+- Returns error if collection fails
+
+**`CompareConfigurations(nodeID1, nodeID2 string) ([]string, error)`**
+- Compares configurations between two nodes
+- Returns list of differences
+- Useful for consistency verification
+
+#### 1.6 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns proxy's own metrics (health, agent count, RPC latency, etc.)
+- Format: Prometheus text format
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics within maintained time window for all agents
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional)
+  - `end_time`: End time for time window (optional)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+#### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	topologyManager     *TopologyManager
+	configCollector     *ConfigurationCollector
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+	nodeID         string
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	sessionID      string
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterRequest with node ID, role, and labels
+- Receives RegisterResponse with session ID and heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in MetricsMessage
+- Sends metrics to Proxy via StreamMetrics() stream
+- Includes node ID, session ID, and timestamp
+- Returns error if retrieval or send fails
+
+**`StartConfigurationStream(ctx context.Context) error`**
+- Establishes bi-directional configuration stream with Proxy
+- Maintains stream for on-demand configuration requests
+- Listens for configuration request commands from Proxy
+- When request received, collects configuration from BanyanDB process and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process when requested by Proxy
+- Finds BanyanDB process (by PID or process name)
+- Reads `/proc/<pid>/cmdline` for command-line flags
+- Reads `/proc/<pid>/environ` for environment variables
+- Sanitizes sensitive data
+- Packages configuration in ConfigurationMessage
+- Sends configuration to Proxy via StreamNodesConfigurations() stream
+- Includes node ID, session ID, and collected timestamp
+- Returns error if collection or send fails
+
+**`SendHeartbeat(ctx context.Context) error`**
+- Sends heartbeat to Proxy
+- Updates connection status
+- Returns error if heartbeat fails
+
+**`Disconnect() error`**
+- Closes connection to Proxy
+- Cleans up resources
+- Returns error if disconnect fails
+
+##### Configuration Flags
+
+**`--proxy-addr`**
+- **Type**: `string`
+- **Default**: `localhost:17900`
+- **Description**: FODC Proxy gRPC address
+
+**`--node-id`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Unique identifier for this BanyanDB node. Must be unique across the cluster. Typically matches the node's identity in the BanyanDB cluster.
+
+**`--node-role`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Role of this BanyanDB node. Valid values: `liaison`, `datanode-hot`, `datanode-warm`, `datanode-cold`, etc. Must match the node's actual role in the cluster.
+
+**`--node-labels`**
+- **Type**: `string` (comma-separated key=value pairs)
+- **Default**: (optional)
+- **Description**: Labels/metadata for this node. Format: `key1=value1,key2=value2`. Examples: `zone=us-west-1,env=production`. Used for filtering and grouping nodes in the Proxy.
+
+**`--heartbeat-interval`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: Interval for sending heartbeats to Proxy. Note: The Proxy may override this value in RegisterResponse.
+
+**`--reconnect-interval`**
+- **Type**: `duration`
+- **Default**: `5s`
+- **Description**: Interval for reconnection attempts when connection to Proxy is lost
+
+#### 2.2 Configuration Collector Component
+
+**Purpose**: Collects configuration from the BanyanDB process running on the same node
+
+##### Core Responsibilities
+
+- **Process Discovery**: Finds the BanyanDB process by PID or process name
+- **Command-Line Collection**: Reads command-line flags from `/proc/<pid>/cmdline`
+- **Environment Collection**: Reads environment variables from `/proc/<pid>/environ`
+- **Data Sanitization**: Sanitizes sensitive data (passwords, tokens, etc.)
+- **Configuration Packaging**: Packages configuration data for transmission to Proxy
+
+##### Core Types
+
+**`ConfigurationCollector`**
+```go
+type ConfigurationCollector struct {
+	banyandbPID    int                    // BanyanDB process ID
+	banyandbProcName string               // BanyanDB process name
+	proxyClient    *ProxyClient
+	logger         *logger.Logger
+}
+```
+
+**`NodeConfiguration`**
+```go
+type NodeConfiguration struct {
+	NodeID        string
+	NodeRole      databasev1.Role
+	StartupParams map[string]string    // Command-line flags from /proc/<pid>/cmdline
+	EnvVars       map[string]string    // Environment variables from /proc/<pid>/environ (sanitized)
+	RuntimeConfig map[string]string    // Runtime configurations (if available)
+	CollectedAt   time.Time
+}
+```
+
+##### Key Functions
+
+**`FindBanyanDBProcess() (int, error)`**
+- Finds BanyanDB process by PID (if known) or process name
+- Searches `/proc` filesystem for matching process
+- Returns process ID or error if not found
+
+**`CollectStartupParams(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/cmdline` file
+- Parses null-separated command-line arguments
+- Extracts flags and values into map
+- Returns error if read fails
+
+**`CollectEnvVars(pid int) (map[string]string, error)`**
+- Reads `/proc/<pid>/environ` file
+- Parses null-separated environment variables
+- Sanitizes sensitive data (passwords, tokens, API keys, etc.)
+- Returns sanitized environment variables map
+- Returns error if read fails
+
+**`CollectConfiguration() (*NodeConfiguration, error)`**
+- Finds BanyanDB process
+- Collects startup parameters from `/proc/<pid>/cmdline`
+- Collects environment variables from `/proc/<pid>/environ`
+- Packages configuration into NodeConfiguration struct
+- Returns error if collection fails
+
+**`SendConfiguration(ctx context.Context) error`**
+- Collects configuration from BanyanDB process
+- Sends configuration to Proxy via gRPC
+- Returns error if send fails
+
+##### Configuration Flags
+
+**`--banyandb-pid`**
+- **Type**: `int`
+- **Default**: (optional, auto-detected if not specified)
+- **Description**: BanyanDB process ID. If not specified, agent will search for process by name.
+
+**`--banyandb-proc-name`**
+- **Type**: `string`
+- **Default**: `banyandb`
+- **Description**: BanyanDB process name to search for if PID is not provided
+
+**`--config-collection-interval`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Description**: Interval for collecting and sending configuration updates to Proxy
+
+## API Design
+
+### gRPC API
+
+#### Service Definition
+
+```protobuf
+syntax = "proto3";
+
+package banyandb.fodc.v1;
+
+import "google/protobuf/timestamp.proto";
+
+service FODCService {
+  // Bi-directional stream for agent registration and heartbeat
+  rpc RegisterAgent(stream RegisterRequest) returns (stream RegisterResponse);
+  
+  // Bi-directional stream for metrics
+  rpc StreamMetrics(stream MetricsMessage) returns (stream MetricsRequest);
+  
+  // Bi-directional stream for node configurations
+  rpc StreamNodesConfigurations(stream ConfigurationMessage) returns (stream ConfigurationRequest);
+}
+
+message RegisterRequest {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> labels = 3;
+}
+
+message RegisterResponse {
+  string session_id = 1;
+  bool success = 2;
+  string message = 3;
+  int64 heartbeat_interval_seconds = 4;
+}
+
+message MetricsMessage {
+  string node_id = 1;
+  string session_id = 2;
+  repeated Metric metrics = 3;
+  google.protobuf.Timestamp timestamp = 4;
+}
+
+message Metric {
+  string name = 1;
+  map<string, string> labels = 2;
+  double value = 3;
+  string description = 4;
+}
+
+message MetricsRequest {
+  google.protobuf.Timestamp start_time = 1;  // Optional start time for time window
+  google.protobuf.Timestamp end_time = 2;    // Optional end time for time window
+}
+
+message ConfigurationMessage {
+  string node_id = 1;
+  string session_id = 2;
+  NodeConfiguration configuration = 3;
+  google.protobuf.Timestamp collected_at = 4;
+}
+
+message NodeConfiguration {
+  string node_id = 1;
+  string node_role = 2;
+  map<string, string> startup_params = 3;  // Command-line flags from /proc/<pid>/cmdline
+  map<string, string> env_vars = 4;         // Environment variables from /proc/<pid>/environ (sanitized)
+  map<string, string> runtime_config = 5;   // Runtime configurations (if available)
+}
+
+message ConfigurationRequest {
+  bool request_all = 1;  // Request configurations from all agents
+  repeated string node_ids = 2;  // Specific node IDs to request (if request_all is false)
+}
+```
+
+### REST API
+
+#### Endpoints
+
+**`GET /metrics`**
+- **Description**: Returns proxy's own metrics in Prometheus format
+- **Response**: Prometheus text format
+- **Example**:
+```
+# HELP fodc_proxy_agents_total Total number of registered agents
+# TYPE fodc_proxy_agents_total gauge
+fodc_proxy_agents_total 5
+
+# HELP fodc_proxy_grpc_requests_total Total gRPC requests
+# TYPE fodc_proxy_grpc_requests_total counter
+fodc_proxy_grpc_requests_total 1234
+```
+
+**`GET /metrics-windows`**
+- **Description**: Returns metrics within time window from all agents
+- **Query Parameters**:
+  - `start_time` (optional): RFC3339 timestamp
+  - `end_time` (optional): RFC3339 timestamp
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- **Response**: Prometheus text format with node metadata labels
+- **Example**:
+```
+# HELP banyandb_stream_tst_inverted_index_total_doc_count Total document count
+# TYPE banyandb_stream_tst_inverted_index_total_doc_count gauge
+banyandb_stream_tst_inverted_index_total_doc_count{index="test",node_id="node1",node_role="datanode-hot"} 12345
+```
+
+**`GET /cluster`**
+- **Description**: Returns cluster topology and node information
+- **Query Parameters**:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role
+- **Response**: JSON
+- **Example**:
+```json
+{
+  "nodes": [
+    {
+      "node_id": "node1",
+      "node_role": "liaison",
+      "address": "192.168.1.1:17900",
+      "labels": {
+        "zone": "us-west-1"
+      },
+      "status": "online",
+      "last_heartbeat": "2024-01-01T12:00:00Z",
+      "runtime_metrics": {
+        "load": 0.5,
+        "health": "healthy"
+      }
+    }
+  ],
+  "updated_at": "2024-01-01T12:00:00Z"
+}
+```
+
+**`GET /cluster/config`**
+- **Description**: Returns node configurations
+- **Query Parameters**:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role
+- **Response**: JSON
+- **Example**:
+```json
+{
+  "configurations": [
+    {
+      "node_id": "node1",
+      "node_role": "liaison",
+      "startup_params": {
+        "--listen-addr": ":17900",
+        "--data-path": "/data"
+      },
+      "runtime_config": {
+        "max_connections": 1000
+      },
+      "collected_at": "2024-01-01T12:00:00Z"
+    }
+  ]
+}
+```
+
+**`GET /health`**
+- **Description**: Health check endpoint
+- **Response**: JSON
+- **Example**:
+```json
+{
+  "status": "healthy",
+  "agents_online": 5,
+  "agents_total": 5,
+  "uptime_seconds": 3600
+}
+```
+
+## Data Flow
+
+### Agent Registration Flow
+
+```
+1. FODC Agent Starts
+   - Reads node information from configuration flags:
+     * --node-id: Unique node identifier
+     * --node-role: Node role (liaison, datanode-hot, etc.)
+     * --node-labels: Optional node labels/metadata
+   ↓
+2. Agent Connects to Proxy via gRPC
+   - Uses --proxy-addr flag to determine Proxy address
+   ↓
+3. Agent Establishes RegisterAgent() Bi-directional Stream
+   - Opens stream to Proxy
+   ↓
+4. Agent Sends RegisterRequest
+   - Sends node ID, role, labels from configuration flags
+   ↓
+5. Proxy Validates Registration
+   - Checks node ID uniqueness
+   - Validates role
+   ↓
+6. Proxy Registers Agent
+   - Creates AgentInfo in AgentRegistry
+   - Updates ClusterTopology
+   ↓
+7. Proxy Sends RegisterResponse via Stream
+   - Session ID for subsequent requests
+   - Heartbeat interval
+   ↓
+8. Agent Maintains Registration Stream
+   - Sends periodic heartbeats via RegisterRequest
+   - Receives RegisterResponse updates
+   - Stream remains open for re-registration if needed
+   ↓
+9. Agent Establishes Metrics Stream
+   - StreamMetrics() for metrics collection
+```
+
+### Metrics Collection Flow
+
+```
+1. External Client Requests Metrics
+   - GET /metrics (proxy metrics)
+   - GET /metrics-windows (agent metrics with time window)
+   ↓
+2. APIServer Receives Request
+   - Routes to metrics handler
+   - Extracts query parameters (start_time, end_time, node_id, role)
+   ↓
+3. Proxy Requests Metrics from All Agents
+   - For each registered agent, sends metrics request via StreamMetrics()
+   - Uses bi-directional stream to request current metrics
+   - May filter by node_id or role if specified in query
+   ↓
+4. Agents Retrieve Metrics from Flight Recorder
+   - Each agent receives metrics request via StreamMetrics()
+   - Agent retrieves metrics from Flight Recorder (in-memory storage)
+   - Flight Recorder continuously collects and stores metrics from:
+     * BanyanDB /metrics endpoint scraping
+     * KTM/OS telemetry collection
+   - Packages metrics in MetricsMessage
+   ↓
+5. Agents Send Metrics to Proxy
+   - Each agent sends MetricsMessage via StreamMetrics()
+   - Includes node ID, timestamp, and collected metrics
+   ↓
+6. Proxy Receives Metrics from All Agents
+   - Collects MetricsMessage from each agent
+   - Waits for responses from all requested agents (with timeout)
+   ↓
+7. Proxy Enriches Metrics
+   - Adds node metadata (role, ID, labels) to each metric
+   - Normalizes metric format across agents
+   - Filters by time window if specified
+   ↓
+8. MetricsAggregator Aggregates Metrics
+   - Combines metrics from all agents
+   - Stores in MetricsWindow for /metrics-windows endpoint
+   - Associates metrics with node ID and role
+   ↓
+9. Proxy Formats and Returns Response
+   - Converts aggregated metrics to Prometheus text format
+   - Includes node metadata labels
+   - Returns HTTP 200 response to external client
+```
+
+### Configuration Collection Flow
+
+```
+1. Agent Establishes Configuration Stream
+   - Agent calls StartConfigurationStream() to establish bi-directional stream
+   - Stream remains open for on-demand configuration requests
+   ↓
+2. External Client Requests Configuration
+   - GET /cluster/config (all configurations)
+   - GET /cluster/config?node_id=X (specific node)
+   - GET /cluster/config?role=Y (role-filtered)
+   ↓
+3. Proxy Requests Configuration from Agents
+   - For each registered agent (or filtered agents), sends configuration request via StreamNodesConfigurations()
+   - Uses bi-directional stream to request current configuration
+   - May filter by node_id or role if specified in query
+   ↓
+4. Agent Collects Configuration from BanyanDB Process
+   - Agent receives configuration request via StreamNodesConfigurations()
+   - Agent calls RetrieveAndSendConfiguration()
+   - Finds BanyanDB process:
+     * By PID (The pod should be configured with `shareProcessNamespace: true` so the agent sidecar can see the target container's `/proc` entries)

Review Comment:
   Inconsistent terminology for BanyanDB process discovery. Line 749 mentions "by PID (if known) or process name" suggesting both methods, but line 1083 only mentions "By PID" with a note about shareProcessNamespace. The documentation should clarify whether process name-based discovery is still supported or if PID-based discovery is the only method, especially given the Kubernetes configuration requirement mentioned in line 1083.
   ```suggestion
        * By PID (recommended in containerized/Kubernetes deployments; the pod should be configured with `shareProcessNamespace: true` so the agent sidecar can see the target container's `/proc` entries)
        * Or by process name, in environments where direct PID-based discovery is not available, by first resolving the PID from the process name before reading `/proc`
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2639591013


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,933 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID            string                    // Unique node identifier
+	NodeRole          databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	PrimaryAddress    Address                  // Primary agent gRPC address
+	SecondaryAddresses map[string]Address      // Secondary addresses with names (e.g., "metrics": Address, "gossip": Address)
+	Labels            map[string]string        // Node labels/metadata
+	RegisteredAt      time.Time                // Registration timestamp
+	LastHeartbeat     time.Time               // Last heartbeat timestamp
+	Status            AgentStatus               // Current agent status
+}
+
+type Address struct {
+	IP   string
+	Port int
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "unconnected"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[AgentKey]*AgentInfo      // Map from agent key (IP+port+role+labels) to agent info
+	mu        sync.RWMutex                 // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+
+type AgentKey struct {
+	IP     string                 // Primary IP address
+	Port   int                    // Primary port
+	Role   databasev1.Role       // Node role
+	Labels map[string]string      // Node labels (used for key matching)
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Creates AgentKey from primary IP + port + role + labels
+- Uses AgentKey as the map key (not nodeID)
+- Validates primary address and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(key AgentKey) error`**
+- Removes agent from registry using AgentKey
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline longer than `--agent-cleanup-timeout` (detected by heartbeat health check)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(key AgentKey) error`**
+- Updates last heartbeat timestamp for agent using AgentKey
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(ip string, port int, role databasev1.Role, labels map[string]string) (*AgentInfo, error)`**
+- Retrieves agent information by primary IP + port + role + labels
+- Creates AgentKey from the provided parameters
+- Looks up agent in registry using AgentKey
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Continuously runs heartbeat health checks regardless of cleanup timeout
+- Unregisters agents that have been offline longer than `--agent-cleanup-timeout`
+- Agents that cannot maintain connection will be removed after the cleanup timeout period
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Minimum**: Must be greater than `--agent-heartbeat-timeout`
+- **Description**: Timeout for automatically unregistering agents that have been offline. Agents that cannot maintain connection will be removed after being offline longer than this timeout. The heartbeat health check continues running regardless of this timeout. This timeout must be greater than `--agent-heartbeat-timeout` to allow for proper health checking.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	logger          *logger.Logger
+}
+
+// Example gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	Key         AgentKey              // Agent key (IP+port+role+labels) for registry lookup
+	NodeID      string                // Node ID (for reference, not used as key)
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent (includes primary address, role, labels)
+- Creates AgentKey from primary IP + port + role + labels
+- Validates registration information
+- Registers agent with AgentRegistry using AgentKey (not nodeID)
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Sends metrics requests from Proxy to agent (on-demand collection)
+- Receives metrics from agent at Proxy in response to requests
+- Proxy initiates by sending a metrics request via MetricsRequest when an external client queries metrics
+- Agent responds with MetricsMessage containing the collected metrics
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Extract agent key (primary IP + port + role + labels) from the connection
+  3. Check if this is the last active stream for the agent
+  4. If all streams are closed, call `AgentRegistry.UnregisterAgent(agentKey)` to clean up
+  5. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration occurs after being offline longer than `--agent-cleanup-timeout`. Heartbeat health checks continue running regardless.
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Time Window Filtering**: Filters metrics by time window when requested by external clients (agents filter based on Flight Recorder data)
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent with time window filter (if specified)
+- Agents retrieve metrics from Flight Recorder (in-memory storage) filtered by time window and respond
+- Waits for MetricsMessage responses from agents (with timeout)
+- Enriches metrics with node metadata from registry
+- Combines metrics from all agents into a single list
+- Returns aggregated metrics (not stored, only returned)
+- Returns error if collection fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with no time filter to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint without time parameters
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with time window filter
+- Agents filter metrics from Flight Recorder by the specified time range
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+##### Configuration Flags
+
+*Note: No configuration flags needed for MetricsAggregator since metrics are collected on-demand and not stored.*
+
+#### 1.4 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns latest metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional) - filters metrics by start time (agents filter from Flight Recorder)
+  - `end_time`: End time for time window (optional) - filters metrics by end time (agents filter from Flight Recorder)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Runtime metrics (optional)

Review Comment:
   No metrics should be included in case of overload. 
   We should have Node relationships.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2640329613


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,941 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID            string                    // Unique node identifier
+	NodeRole          databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	PrimaryAddress    Address                  // Primary agent gRPC address
+	SecondaryAddresses map[string]Address      // Secondary addresses with names (e.g., "metrics": Address, "gossip": Address)
+	Labels            map[string]string        // Node labels/metadata
+	RegisteredAt      time.Time                // Registration timestamp
+	LastHeartbeat     time.Time               // Last heartbeat timestamp
+	Status            AgentStatus               // Current agent status
+}
+
+type Address struct {
+	IP   string
+	Port int
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "unconnected"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[AgentKey]*AgentInfo      // Map from agent key (IP+port+role+labels) to agent info
+	mu        sync.RWMutex                 // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+
+type AgentKey struct {
+	IP     string                 // Primary IP address
+	Port   int                    // Primary port
+	Role   databasev1.Role       // Node role
+	Labels map[string]string      // Node labels (used for key matching)
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Creates AgentKey from primary IP + port + role + labels
+- Uses AgentKey as the map key (not nodeID)
+- Validates primary address and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(key AgentKey) error`**
+- Removes agent from registry using AgentKey
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline longer than `--agent-cleanup-timeout` (detected by heartbeat health check)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(key AgentKey) error`**
+- Updates last heartbeat timestamp for agent using AgentKey
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(ip string, port int, role databasev1.Role, labels map[string]string) (*AgentInfo, error)`**
+- Retrieves agent information by primary IP + port + role + labels
+- Creates AgentKey from the provided parameters
+- Looks up agent in registry using AgentKey
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Continuously runs heartbeat health checks regardless of cleanup timeout
+- Unregisters agents that have been offline longer than `--agent-cleanup-timeout`
+- Agents that cannot maintain connection will be removed after the cleanup timeout period
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Minimum**: Must be greater than `--agent-heartbeat-timeout`
+- **Description**: Timeout for automatically unregistering agents that have been offline. Agents that cannot maintain connection will be removed after being offline longer than this timeout. The heartbeat health check continues running regardless of this timeout. This timeout must be greater than `--agent-heartbeat-timeout` to allow for proper health checking.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	logger          *logger.Logger
+}
+
+// Example gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterAgentRequest) (stream RegisterAgentResponse) error
+// StreamMetrics(stream StreamMetricsRequest) (stream StreamMetricsResponse) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	Key         AgentKey              // Agent key (IP+port+role+labels) for registry lookup
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent (includes primary address, role, labels)
+- Creates AgentKey from primary IP + port + role + labels
+- Validates registration information
+- Registers agent with AgentRegistry using AgentKey
+- Sends registration responses
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Sends metrics requests from Proxy to agent (on-demand collection)
+- Receives metrics from agent at Proxy in response to requests
+- Proxy initiates by sending a metrics request via StreamMetricsResponse when an external client queries metrics
+- Agent responds with StreamMetricsRequest containing the collected metrics
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Extract agent key (primary IP + port + role + labels) from the connection
+  3. Check if this is the last active stream for the agent
+  4. If all streams are closed, call `AgentRegistry.UnregisterAgent(agentKey)` to clean up
+  5. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration occurs after being offline longer than `--agent-cleanup-timeout`. Heartbeat health checks continue running regardless.
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Time Window Filtering**: Filters metrics by time window when requested by external clients (agents filter based on Flight Recorder data)
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent with time window filter (if specified)
+- Agents retrieve metrics from Flight Recorder (in-memory storage) filtered by time window and respond
+- Waits for StreamMetricsRequest responses from agents (with timeout)
+- Identifies agent connection from stream context and looks up AgentKey
+- Enriches metrics with node metadata (IP, port, role, labels) from AgentKey
+- Combines metrics from all agents into a single list
+- Returns aggregated metrics (not stored, only returned)
+- Returns error if collection fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with no time filter to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint without time parameters
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with time window filter
+- Agents filter metrics from Flight Recorder by the specified time range
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+##### Configuration Flags
+
+*Note: No configuration flags needed for MetricsAggregator since metrics are collected on-demand and not stored.*
+
+#### 1.4 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns latest metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format

Review Comment:
   This can't be the Prometheus text format. Because as a window, it has time series values for each metric. We need a JSON or YAML format for this values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638967099


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.

Review Comment:
   This can't be `0`. If the agent can't maintain the connection, it should be removed. Healthy check heart-beat should keep running.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng merged PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638992019


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Storage**: Stores aggregated metrics with time-series context

Review Comment:
   No storage or permanent memory cost is expected in the FODC proxy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2638989946


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,1278 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID      string                    // Unique node identifier
+	NodeRole    databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	Address     string                    // Agent gRPC address
+	Labels      map[string]string        // Node labels/metadata
+	RegisteredAt time.Time                // Registration timestamp
+	LastHeartbeat time.Time               // Last heartbeat timestamp
+	Status      AgentStatus               // Current agent status
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "offline"
+	AgentStatusUnknown AgentStatus = "unknown"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[string]*AgentInfo      // Map from node ID to agent info
+	mu        sync.RWMutex               // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Validates node ID and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(nodeID string) error`**
+- Removes agent from registry
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline for extended period (cleanup after heartbeat timeout)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(nodeID string) error`**
+- Updates last heartbeat timestamp for agent
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(nodeID string) (*AgentInfo, error)`**
+- Retrieves agent information by node ID
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Optionally unregisters agents that have been offline for extended period (if `--agent-cleanup-timeout` is configured)
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `0` (disabled, agents are not auto-unregistered)
+- **Description**: Timeout for automatically unregistering agents that have been offline. If set to 0, agents remain registered even when offline. If set to a positive duration, agents offline longer than this timeout will be unregistered.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	configCollector *ConfigurationCollector
+	logger          *logger.Logger
+}
+
+// gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterRequest) (stream RegisterResponse) error
+// StreamMetrics(stream MetricsMessage) (stream MetricsRequest) error
+// StreamNodesConfigurations(stream ConfigurationMessage) (stream ConfigurationRequest) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent
+- Validates registration information
+- Registers agent with AgentRegistry
+- Sends registration responses with assigned session ID
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Receives metrics requests from Proxy (on-demand collection)
+- Sends metrics from agent to Proxy when requested
+- Proxy sends metrics request via MetricsRequest when external client queries metrics
+- Agent responds with MetricsMessage containing collected metrics
+- Manages stream lifecycle
+
+**`StreamNodesConfigurations(stream FODCService_StreamNodesConfigurationsServer) error`**
+- Handles bi-directional configuration streaming
+- Receives configuration requests from Proxy (on-demand collection)
+- Sends configuration from agent to Proxy when requested
+- Proxy sends configuration request via ConfigurationRequest when external client queries configurations
+- Agent responds with ConfigurationMessage containing collected configuration
+- Forwards configuration to ConfigurationCollector for storage
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Check if this is the last active stream for the agent
+  3. If all streams are closed, call `AgentRegistry.UnregisterAgent()` to clean up
+  4. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration may occur after extended offline period (configurable)
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Time Window Management**: Maintains sliding window of metrics for `/metrics-windows` endpoint

Review Comment:
   I think we don't need to maintain anything or cache anything. All data should be fetched from the data node on-demand only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2640333835


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,941 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities
+
+**FODC Proxy Core Responsibilities**
+- Accept bi-directional gRPC connections from FODC Agents
+- Register and track agent lifecycle (online/offline, heartbeat monitoring)
+- Aggregate metrics from all agents with node metadata enrichment
+- Maintain cluster topology view based on agent registrations
+- Collect and expose node configurations for audit and consistency checks
+- Expose unified REST/Prometheus-style interfaces for external consumption
+- Provide proxy-level metrics (health, agent count, RPC latency, etc.)
+
+## Component Design
+
+### 1. Proxy Components
+
+#### 1.1 Agent Registry Component
+
+**Purpose**: Manages the lifecycle and state of all connected FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Registration**: Accepts agent registration requests via gRPC
+- **Health Monitoring**: Tracks agent heartbeat and connection status
+- **State Management**: Maintains agent state (online/offline, last heartbeat time)
+- **Topology Building**: Aggregates agent registrations into cluster topology view
+- **Connection Management**: Handles connection failures, reconnections, and cleanup
+
+##### Core Types
+
+**`AgentInfo`**
+```go
+type AgentInfo struct {
+	NodeID            string                    // Unique node identifier
+	NodeRole          databasev1.Role          // Node role (liaison, datanode-hot, etc.)
+	PrimaryAddress    Address                  // Primary agent gRPC address
+	SecondaryAddresses map[string]Address      // Secondary addresses with names (e.g., "metrics": Address, "gossip": Address)
+	Labels            map[string]string        // Node labels/metadata
+	RegisteredAt      time.Time                // Registration timestamp
+	LastHeartbeat     time.Time               // Last heartbeat timestamp
+	Status            AgentStatus               // Current agent status
+}
+
+type Address struct {
+	IP   string
+	Port int
+}
+
+type AgentStatus string
+
+const (
+	AgentStatusOnline  AgentStatus = "online"
+	AgentStatusOffline AgentStatus = "unconnected"
+)
+```
+
+**`AgentRegistry`**
+```go
+type AgentRegistry struct {
+	agents    map[AgentKey]*AgentInfo      // Map from agent key (IP+port+role+labels) to agent info
+	mu        sync.RWMutex                 // Protects agents map
+	logger    *logger.Logger
+	heartbeatTimeout time.Duration        // Timeout for considering agent offline
+}
+
+type AgentKey struct {
+	IP     string                 // Primary IP address
+	Port   int                    // Primary port
+	Role   databasev1.Role       // Node role
+	Labels map[string]string      // Node labels (used for key matching)
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(ctx context.Context, info *AgentInfo) error`**
+- Registers a new agent or updates existing agent information
+- Creates AgentKey from primary IP + port + role + labels
+- Uses AgentKey as the map key (not nodeID)
+- Validates primary address and role
+- Updates topology view
+- Returns error if registration fails
+
+**`UnregisterAgent(key AgentKey) error`**
+- Removes agent from registry using AgentKey
+- Cleans up associated resources
+- Updates topology view
+- Called in the following scenarios:
+  - When agent's registration stream closes (connection lost)
+  - When agent's all streams are closed and connection is terminated
+  - When agent has been offline longer than `--agent-cleanup-timeout` (detected by heartbeat health check)
+  - During graceful shutdown or manual agent removal
+  - When agent explicitly requests unregistration via stream
+
+**`UpdateHeartbeat(key AgentKey) error`**
+- Updates last heartbeat timestamp for agent using AgentKey
+- Marks agent as online if it was offline
+- Returns error if agent not found
+
+**`GetAgent(ip string, port int, role databasev1.Role, labels map[string]string) (*AgentInfo, error)`**
+- Retrieves agent information by primary IP + port + role + labels
+- Creates AgentKey from the provided parameters
+- Looks up agent in registry using AgentKey
+- Returns error if agent not found
+
+**`ListAgents() []*AgentInfo`**
+- Returns list of all registered agents
+- Thread-safe read operation
+
+**`ListAgentsByRole(role databasev1.Role) []*AgentInfo`**
+- Returns agents filtered by role
+- Useful for role-specific operations
+
+**`CheckAgentHealth() error`**
+- Periodically checks agent health based on heartbeat timeout
+- Marks agents as offline if heartbeat timeout exceeded
+- Continuously runs heartbeat health checks regardless of cleanup timeout
+- Unregisters agents that have been offline longer than `--agent-cleanup-timeout`
+- Agents that cannot maintain connection will be removed after the cleanup timeout period
+- Returns aggregated health status
+
+##### Configuration Flags
+
+**`--agent-heartbeat-timeout`**
+- **Type**: `duration`
+- **Default**: `30s`
+- **Description**: Timeout for considering an agent offline if no heartbeat received
+
+**`--max-agents`**
+- **Type**: `int`
+- **Default**: `1000`
+- **Description**: Maximum number of agents that can be registered
+
+**`--agent-cleanup-timeout`**
+- **Type**: `duration`
+- **Default**: `5m`
+- **Minimum**: Must be greater than `--agent-heartbeat-timeout`
+- **Description**: Timeout for automatically unregistering agents that have been offline. Agents that cannot maintain connection will be removed after being offline longer than this timeout. The heartbeat health check continues running regardless of this timeout. This timeout must be greater than `--agent-heartbeat-timeout` to allow for proper health checking.
+
+#### 1.2 gRPC Server Component
+
+**Purpose**: Handles bi-directional gRPC communication with FODC Agents
+
+##### Core Responsibilities
+
+- **Agent Connection Handling**: Accepts and manages gRPC connections from agents
+- **Streaming Support**: Supports bi-directional streaming for metrics
+- **Protocol Implementation**: Implements FODC gRPC service protocol
+- **Connection Lifecycle**: Manages connection establishment, maintenance, and cleanup
+
+##### Core Types
+
+**`FODCService`** (gRPC Service Implementation)
+```go
+type FODCService struct {
+	registry        *AgentRegistry
+	metricsAggregator *MetricsAggregator
+	logger          *logger.Logger
+}
+
+// Example gRPC service methods (to be defined in proto)
+// RegisterAgent(stream RegisterAgentRequest) (stream RegisterAgentResponse) error
+// StreamMetrics(stream StreamMetricsRequest) (stream StreamMetricsResponse) error
+```
+
+**`AgentConnection`**
+```go
+type AgentConnection struct {
+	Key         AgentKey              // Agent key (IP+port+role+labels) for registry lookup
+	NodeID      string
+	Stream      grpc.ServerStream
+	Context     context.Context
+	Cancel      context.CancelFunc
+	LastActivity time.Time
+}
+```
+
+##### Key Functions
+
+**`RegisterAgent(stream FODCService_RegisterAgentServer) error`**
+- Handles bi-directional agent registration stream
+- Receives registration requests from agent (includes primary address, role, labels)
+- Creates AgentKey from primary IP + port + role + labels
+- Validates registration information
+- Registers agent with AgentRegistry using AgentKey
+- Sends registration responses
+- Maintains stream for heartbeat and re-registration
+
+**`StreamMetrics(stream FODCService_StreamMetricsServer) error`**
+- Handles bi-directional metrics streaming
+- Sends metrics requests from Proxy to agent (on-demand collection)
+- Receives metrics from agent at Proxy in response to requests
+- Proxy initiates by sending a metrics request via StreamMetricsResponse when an external client queries metrics
+- Agent responds with StreamMetricsRequest containing the collected metrics
+- Manages stream lifecycle
+
+##### Connection Lifecycle Management
+
+**Stream Closure Handling**
+- When a stream closes (due to network error, agent shutdown, or timeout), the gRPC server should:
+  1. Detect stream closure via context cancellation or stream error
+  2. Extract agent key (primary IP + port + role + labels) from the connection
+  3. Check if this is the last active stream for the agent
+  4. If all streams are closed, call `AgentRegistry.UnregisterAgent(agentKey)` to clean up
+  5. Update topology to reflect agent offline status
+
+**Graceful vs. Ungraceful Disconnection**
+- **Graceful**: Agent sends explicit disconnect message before closing stream → immediate unregistration
+- **Ungraceful**: Stream closes unexpectedly → unregistration happens after detecting all streams closed
+- **Heartbeat Timeout**: Agent marked offline by `CheckAgentHealth()` → unregistration occurs after being offline longer than `--agent-cleanup-timeout`. Heartbeat health checks continue running regardless.
+
+##### Configuration Flags
+
+**`--grpc-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17900`
+- **Description**: gRPC server address where the Proxy listens for agent connections
+
+**`--grpc-max-msg-size`**
+- **Type**: `int`
+- **Default**: `4194304` (4MB)
+- **Description**: Maximum message size for gRPC messages
+
+#### 1.3 Metrics Aggregator Component
+
+**Purpose**: Aggregates and enriches metrics from all agents
+
+##### Core Responsibilities
+
+- **On-Demand Metrics Collection**: Requests metrics from agents via gRPC streams when external clients query metrics
+- **Metrics Request Coordination**: Coordinates metrics requests to multiple agents concurrently
+- **Metadata Enrichment**: Adds node metadata (role, ID, labels) to metrics
+- **Normalization**: Normalizes metric formats and labels across agents
+- **Time Window Filtering**: Filters metrics by time window when requested by external clients (agents filter based on Flight Recorder data)
+
+##### Core Types
+
+**`AggregatedMetric`**
+```go
+type AggregatedMetric struct {
+	Name        string                 // Metric name
+	Labels      map[string]string      // Metric labels (including node metadata)
+	Value       float64                // Metric value
+	Timestamp   time.Time              // Metric timestamp
+	NodeID      string                 // Source node ID
+	NodeRole    databasev1.Role       // Source node role
+	Description string                 // Metric description/HELP text
+}
+```
+
+**`MetricsAggregator`**
+```go
+type MetricsAggregator struct {
+	registry      *AgentRegistry
+	grpcService   *FODCService        // For requesting metrics from agents
+	mu            sync.RWMutex
+	logger        *logger.Logger
+}
+
+type MetricsFilter struct {
+	NodeIDs []string              // Filter by specific node IDs (empty = all nodes)
+	Role    databasev1.Role       // Filter by node role (optional)
+	StartTime *time.Time          // Start time for time window (optional)
+	EndTime   *time.Time          // End time for time window (optional)
+}
+```
+
+##### Key Functions
+
+**`CollectMetricsFromAgents(ctx context.Context, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Requests metrics from all agents (or filtered agents) when external client queries
+- Sends metrics request via StreamMetrics() to each agent with time window filter (if specified)
+- Agents retrieve metrics from Flight Recorder (in-memory storage) filtered by time window and respond
+- Waits for StreamMetricsRequest responses from agents (with timeout)
+- Identifies agent connection from stream context and looks up AgentKey
+- Enriches metrics with node metadata (IP, port, role, labels) from AgentKey
+- Combines metrics from all agents into a single list
+- Returns aggregated metrics (not stored, only returned)
+- Returns error if collection fails
+
+**`GetLatestMetrics(ctx context.Context) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with no time filter to get current metrics
+- Returns latest metrics from all agents
+- Used for `/metrics-windows` endpoint without time parameters
+- Returns error if collection fails
+
+**`GetMetricsWindow(ctx context.Context, startTime, endTime time.Time, filter *MetricsFilter) ([]*AggregatedMetric, error)`**
+- Triggers on-demand collection from all agents
+- Calls CollectMetricsFromAgents() with time window filter
+- Agents filter metrics from Flight Recorder by the specified time range
+- Returns metrics within specified time range
+- Used for `/metrics-windows` endpoint with time parameters
+- Returns error if collection fails
+
+##### Configuration Flags
+
+*Note: No configuration flags needed for MetricsAggregator since metrics are collected on-demand and not stored.*
+
+#### 1.4 HTTP/REST API Server Component
+
+**Purpose**: Exposes REST and Prometheus-style endpoints for external consumption
+
+##### Core Responsibilities
+
+- **REST API**: Implements REST endpoints for cluster topology and configuration
+- **Prometheus Integration**: Exposes Prometheus-compatible metrics endpoints
+- **Request Handling**: Handles HTTP requests and routes to appropriate handlers
+- **Response Formatting**: Formats responses in appropriate formats (JSON, Prometheus text)
+
+##### API Endpoints
+
+**`GET /metrics`**
+- Returns latest metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- Used by: Prometheus scrapers, monitoring systems
+
+**`GET /metrics-windows`**
+- Returns metrics from all agents (on-demand collection, not stored in Proxy)
+- Includes node metadata (role, ID, labels)
+- Format: Prometheus text format
+- Query parameters:
+  - `start_time`: Start time for time window (optional) - filters metrics by start time (agents filter from Flight Recorder)
+  - `end_time`: End time for time window (optional) - filters metrics by end time (agents filter from Flight Recorder)
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /cluster`**
+- Returns cluster topology and node status
+- Format: JSON
+- Response includes:
+  - List of all registered nodes
+  - Node identity (ID, name, address)
+  - Node role
+  - Node labels
+  - Agent status (online/offline, last heartbeat)
+  - Node relationships
+- Note: Node information is obtained from agent registration data stored in AgentRegistry
+
+**`GET /cluster/config`**
+- Returns node configurations
+- Format: JSON
+- Query parameters:
+  - `node_id`: Filter by node ID (optional)
+  - `role`: Filter by node role (optional)
+
+**`GET /health`**
+- Health check endpoint
+- Format: JSON
+- Returns proxy health status
+
+##### Core Types
+
+**`APIServer`**
+```go
+type APIServer struct {
+	metricsAggregator   *MetricsAggregator
+	server              *http.Server
+	logger              *logger.Logger
+}
+```
+
+##### Key Functions
+
+**`Start(listenAddr string) error`**
+- Starts HTTP server
+- Registers all API endpoints
+- Returns error if start fails
+
+**`Stop() error`**
+- Gracefully stops HTTP server
+- Waits for in-flight requests
+- Returns error if stop fails
+
+##### Configuration Flags
+
+**`--http-listen-addr`**
+- **Type**: `string`
+- **Default**: `:17901`
+- **Description**: HTTP server listen address
+
+**`--http-read-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP read timeout
+
+**`--http-write-timeout`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: HTTP write timeout
+
+### 2. Agent Components
+
+**Purpose**: Components that run within FODC Agents to communicate with the Proxy
+
+#### 2.1 Proxy Client Component
+
+**Purpose**: Manages connection and communication with the FODC Proxy
+
+##### Core Responsibilities
+
+- **Connection Management**: Establishes and maintains gRPC connection to Proxy
+- **Registration**: Registers agent with Proxy on startup
+- **Heartbeat Management**: Sends periodic heartbeats to maintain connection
+- **Stream Management**: Manages bi-directional streams for metrics
+- **Reconnection Logic**: Handles connection failures and automatic reconnection
+
+##### Core Types
+
+**`ProxyClient`**
+```go
+type ProxyClient struct {
+	proxyAddr      string
+  nodeIP         string
+  nodePort       int
+	nodeRole       databasev1.Role
+	labels         map[string]string
+	conn           *grpc.ClientConn
+	client         fodcv1.FODCServiceClient
+	heartbeatTicker *time.Ticker
+	mu             sync.RWMutex
+	logger         *logger.Logger
+}
+
+type MetricsRequestFilter struct {
+	StartTime *time.Time  // Start time for time window (optional)
+	EndTime   *time.Time  // End time for time window (optional)
+}
+```
+
+**Note**: The ProxyClient integrates with Flight Recorder, which continuously collects and stores metrics in-memory. When the Proxy requests metrics, the agent retrieves them from Flight Recorder rather than collecting them on-demand.
+
+##### Key Functions
+
+**`Connect(ctx context.Context) error`**
+- Establishes gRPC connection to Proxy
+- Returns error if connection fails
+
+**`StartRegistrationStream(ctx context.Context) error`**
+- Establishes bi-directional registration stream with Proxy
+- Sends initial RegisterAgentRequest with primary address (IP + port), role, and labels
+- Receives RegisterAgentResponse with heartbeat interval
+- Maintains stream for periodic heartbeat and re-registration
+- Returns error if stream establishment fails
+
+**`StartMetricsStream(ctx context.Context) error`**
+- Establishes bi-directional metrics stream with Proxy
+- Maintains stream for on-demand metrics requests
+- Listens for metrics request commands from Proxy
+- When request received, retrieves metrics from Flight Recorder and sends response
+- Returns error if stream fails
+
+**`RetrieveAndSendMetrics(ctx context.Context, filter *MetricsRequestFilter) error`**
+- Retrieves metrics from Flight Recorder when requested by Proxy
+- Queries Flight Recorder (in-memory storage) for metrics
+- Flight Recorder contains continuously collected metrics from:
+  * BanyanDB /metrics endpoint scraping
+  * KTM/OS telemetry collection
+  * Other observability sources
+- Filters metrics by time window if specified in request
+- Packages metrics in StreamMetricsRequest (includes metrics and timestamp)
+- Sends metrics to Proxy via StreamMetrics() stream
+- Returns error if retrieval or send fails
+
+**`SendHeartbeat(ctx context.Context) error`**
+- Sends heartbeat to Proxy
+- Updates connection status
+- Returns error if heartbeat fails
+
+**`Disconnect() error`**
+- Closes connection to Proxy
+- Cleans up resources
+- Returns error if disconnect fails
+
+##### Configuration Flags
+
+**`--proxy-addr`**
+- **Type**: `string`
+- **Default**: `localhost:17900`
+- **Description**: FODC Proxy gRPC address
+
+**`--node-ip`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: IP address for this BanyanDB node's primary gRPC address. Used as part of AgentKey for registry identification.
+
+**`--node-port`**
+- **Type**: `int`
+- **Default**: (required, no default)
+- **Description**: Port number for this BanyanDB node's primary gRPC address. Used as part of AgentKey for registry identification.
+
+**`--node-role`**
+- **Type**: `string`
+- **Default**: (required, no default)
+- **Description**: Role of this BanyanDB node. Valid values: `liaison`, `datanode-hot`, `datanode-warm`, `datanode-cold`, etc. Must match the node's actual role in the cluster.
+
+**`--node-labels`**
+- **Type**: `string` (comma-separated key=value pairs)
+- **Default**: (optional)
+- **Description**: Labels/metadata for this node. Format: `key1=value1,key2=value2`. Examples: `zone=us-west-1,env=production`. Used for filtering and grouping nodes in the Proxy.
+
+**`--heartbeat-interval`**
+- **Type**: `duration`
+- **Default**: `10s`
+- **Description**: Interval for sending heartbeats to Proxy. Note: The Proxy may override this value in RegisterAgentResponse.
+
+**`--reconnect-interval`**
+- **Type**: `duration`
+- **Default**: `5s`
+- **Description**: Interval for reconnection attempts when connection to Proxy is lost
+
+## API Design
+
+### gRPC API
+
+#### Service Definition
+
+```protobuf
+
+syntax = "proto3";
+
+package banyandb.fodc.v1;
+
+import "google/protobuf/timestamp.proto";
+
+option go_package = "github.com/apache/skywalking-banyandb/api/proto/banyandb/fodc/v1";
+
+service FODCService {
+  // Bi-directional stream for agent registration and heartbeat
+  rpc RegisterAgent(stream RegisterAgentRequest) returns (stream RegisterAgentResponse);
+  
+  // Bi-directional stream for metrics
+  // Agent sends StreamMetricsRequest (metrics data), Proxy sends StreamMetricsResponse (metrics requests)
+  rpc StreamMetrics(stream StreamMetricsRequest) returns (stream StreamMetricsResponse);
+}
+
+message RegisterAgentRequest {
+  string node_role = 1;
+  map<string, string> labels = 2;
+  Address primary_address = 3;
+  map<string, Address> secondary_addresses = 4;
+}
+
+message Address {
+  string ip = 1;
+  int32 port = 2;
+}
+
+message RegisterAgentResponse {
+  bool success = 1;
+  string message = 2;
+  int64 heartbeat_interval_seconds = 3;
+}
+
+message StreamMetricsRequest {
+  repeated Metric metrics = 1;
+  google.protobuf.Timestamp timestamp = 2;
+}
+
+message Metric {
+  string name = 1;
+  map<string, string> labels = 2;
+  double value = 3;
+  string description = 4;
+}
+
+message StreamMetricsResponse {
+  google.protobuf.Timestamp start_time = 1;  // Optional start time for time window
+  google.protobuf.Timestamp end_time = 2;    // Optional end time for time window
+}
+```
+
+### REST API
+
+#### Endpoints
+
+**`GET /metrics`**
+- **Description**: Returns latest metrics from all agents (on-demand collection, not stored in Proxy)
+- **Query Parameters**:
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- **Response**: Prometheus text format with node metadata labels
+- **Note**: Metrics are collected on-demand from agents' Flight Recorder. No metrics are stored in the Proxy.
+- **Example**:
+```
+# HELP banyandb_stream_tst_inverted_index_total_doc_count Total document count
+# TYPE banyandb_stream_tst_inverted_index_total_doc_count gauge
+banyandb_stream_tst_inverted_index_total_doc_count{index="test",node_id="node1",node_role="datanode-hot"} 12345
+```
+
+**`GET /metrics-windows`**
+- **Description**: Returns metrics from all agents (on-demand collection, not stored in Proxy)
+- **Query Parameters**:
+  - `start_time` (optional): RFC3339 timestamp - filters metrics by start time (agents filter from Flight Recorder)
+  - `end_time` (optional): RFC3339 timestamp - filters metrics by end time (agents filter from Flight Recorder)
+  - `node_id` (optional): Filter by node ID
+  - `role` (optional): Filter by role (liaison, datanode-hot, etc.)
+- **Response**: Prometheus text format with node metadata labels
+- **Note**: Metrics are collected on-demand from agents' Flight Recorder. No metrics are stored in the Proxy.
+- **Example**:
+```
+# HELP banyandb_stream_tst_inverted_index_total_doc_count Total document count
+# TYPE banyandb_stream_tst_inverted_index_total_doc_count gauge
+banyandb_stream_tst_inverted_index_total_doc_count{index="test",node_id="node1",node_role="datanode-hot"} 12345
+```
+
+**`GET /cluster`**
+- **Description**: Returns cluster topology and node information

Review Comment:
   NOTE, for the initial version, we only have the node list, no topology.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs: document the FODC Proxy Development Design [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #911:
URL: https://github.com/apache/skywalking-banyandb/pull/911#discussion_r2639579992


##########
docs/design/fodc/proxy.md:
##########
@@ -0,0 +1,933 @@
+# FODC Proxy Development Design
+
+## Overview
+
+The FODC Proxy is the central control plane and data aggregator for the First Occurrence Data Collection (FODC) infrastructure. It acts as a unified gateway that aggregates observability data from multiple FODC Agents (each co-located with a BanyanDB node) and exposes ecosystem-friendly interfaces to external systems such as Prometheus and other observability platforms.
+
+The Proxy provides:
+
+1. **Agent Management**: Registration, health monitoring, and lifecycle management of FODC Agents
+2. **Metrics Aggregation**: Collects and aggregates metrics from all agents with enriched metadata
+3. **Cluster Topology**: Maintains an up-to-date view of cluster topology, roles, and node states
+4. **Configuration Collection**: Aggregates and exposes node configurations for consistency verification
+
+### Responsibilities

Review Comment:
   No authN design is required in first version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org