Re: [PR] Add lifecycle management [skywalking-banyandb]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on code in PR #618:
URL: https://github.com/apache/skywalking-banyandb/pull/618#discussion_r1988244434


##########
docs/operation/lifecycle.md:
##########
@@ -0,0 +1,213 @@
+# Lifecycle Management in BanyanDB
+
+BanyanDB uses a hot-warm-cold data tiering strategy to optimize storage costs and query performance. The lifecycle management feature automatically migrates data between storage tiers based on configured policies.
+
+## Overview
+
+A dedicated lifecycle agent (an independent Go process) is responsible for:
+
+- Identifying data eligible for migration based on lifecycle stages.
+- Extracting data from the source node.
+- Migrating data to designated target nodes.
+- Removing the original data after a successful migration.
+
+This process ensures data flows correctly through hot, warm, and cold stages as defined in your group configurations.
+
+## Data Migration Process
+
+The lifecycle agent performs the following steps:
+
+1. **Identify Groups for Migration:**
+   - Connect to the metadata service and list all groups.
+   - Filter groups configured with multiple lifecycle stages.
+   - Skip groups already processed (using progress tracking).
+
+2. **Take Snapshots:**
+   - Request a snapshot from the source data node.
+   - Retrieve the stream and measure data directory paths.
+
+3. **Setup Query Services:**
+   - Create read-only services for streams and measures using the snapshot paths.
+
+4. **Process Each Group:**
+   - Determine the current stage based on node labels.
+   - Identify target nodes for the next lifecycle stage.
+   - For each stream or measure:
+      - Query data from the current stage.
+      - Transform data into write requests.
+      - Send write requests to target nodes.
+   - Delete the source data after a successful migration.
+
+5. **Progress Tracking:**
+   - Track migration progress in a persistent progress file.
+   - Enable crash recovery by resuming from the last saved state.
+   - Clean up the progress file once the migration is complete.
+
+## Configuration
+
+Lifecycle stages are defined within the group configuration using the `LifecycleStage` structure. 
+
+| Field         | Description                                                          |
+| ------------- | -------------------------------------------------------------------- |
+| `name`        | Stage name (e.g., "hot", "warm", "cold")                             |
+| `shard_num`   | Number of shards allocated for this stage                            |
+| `segment_interval` | Time interval for data segmentation (uses `IntervalRule`)       |
+| `ttl`         | Time-to-live before data moves to the next stage (uses `IntervalRule`)|
+| `node_selector` | Label selector to identify target nodes for this stage             |
+| `close`       | Indicates whether to close segments that are no longer live          |

Review Comment:
   Does this have a default value? We should keep `false` by default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org