You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by ha...@apache.org on 2025/08/01 22:37:51 UTC
(skywalking-banyandb) 04/06: Implement file-based measure migration and enhance progress tracking
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch lifecyc-sync
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 1bf0ab22f2cc9d81581947e0482b6d8d1fd0f522
Author: Gao Hongtao <ha...@gmail.com>
AuthorDate: Fri Aug 1 17:55:40 2025 +0800
Implement file-based measure migration and enhance progress tracking
- Introduced `measure_migration_visitor.go` and `migration_integration.go` to facilitate file-based measure migration with progress tracking.
- Refactored `lifecycleService` to utilize the new file-based migration approach for measures, enhancing efficiency and reducing overhead.
---
.../backup/lifecycle/MEASURE_MIGRATION_DESIGN.md | 420 --------------
...ion_visitor.go => measure_migration_visitor.go} | 245 ++------
...ion_integration.go => migration_integration.go} | 74 +++
banyand/backup/lifecycle/progress.go | 317 ++++++-----
banyand/backup/lifecycle/progress_test.go | 145 -----
banyand/backup/lifecycle/service.go | 367 +++++++-----
banyand/backup/lifecycle/steps.go | 158 ------
banyand/backup/lifecycle/steps_test.go | 625 ---------------------
...tion_visitor.go => stream_migration_visitor.go} | 12 +-
banyand/measure/part.go | 75 +++
banyand/measure/part_metadata.go | 24 +
banyand/measure/visitor.go | 90 +++
12 files changed, 709 insertions(+), 1843 deletions(-)
diff --git a/banyand/backup/lifecycle/MEASURE_MIGRATION_DESIGN.md b/banyand/backup/lifecycle/MEASURE_MIGRATION_DESIGN.md
deleted file mode 100644
index b2e03016..00000000
--- a/banyand/backup/lifecycle/MEASURE_MIGRATION_DESIGN.md
+++ /dev/null
@@ -1,420 +0,0 @@
-# File-Based Measure Migration Design
-
-## Overview
-
-This document outlines the design for refactoring measure migration from a query-based approach to a file-based streaming approach, following the successful pattern established in stream migration.
-
-## Current Architecture Analysis
-
-### Current Query-Based Approach
-- **File**: `steps.go:230-296` (`migrateMeasure` function)
-- **Method**: Individual record querying via `MeasureQueryOptions`
-- **Transfer**: Message-based publishing via `queue.BatchPublisher`
-- **Granularity**: Per-record processing
-- **Performance**: High overhead due to individual record serialization
-
-### Target File-Based Approach
-- **Pattern**: Similar to `file_migration_visitor.go` stream migration
-- **Method**: Direct file streaming of measure parts
-- **Transfer**: Chunked streaming via `queue.ChunkedSyncClient`
-- **Granularity**: Per-part file processing
-- **Performance**: Bulk file transfer optimization
-
-## Measure Storage Architecture
-
-Based on analysis of `measure/metadata.go` and `measure/tstable.go`, measures are stored at **group level**, not per-measure:
-
-### Group-Level Storage Structure
-```
-measure_data/
-└── <group_name>/ # Group directory (e.g., "sw_metric")
- ├── 000000000000abc1/ # Part directory (epoch-based ID)
- │ ├── metadata.json # Part metadata
- │ ├── primary.bin # Primary index
- │ ├── meta.bin # Block metadata (compressed)
- │ ├── timestamps.bin # Timestamp data
- │ ├── fv.bin # Field values
- │ ├── <tagfamily>.tf # Tag family data
- │ └── <tagfamily>.tfm # Tag family metadata
- ├── 000000000000abc2/ # Another part directory
- └── snapshot_xyz.snp # Snapshot files
-```
-
-### Key Architecture Points
-1. **Group-Level Storage**: All measures in a group share the same TSDB storage (`path.Join(s.path, group)`)
-2. **Single TSDB per Group**: `g.SupplyTSDB()` returns one TSDB instance per group
-3. **Part-Based Organization**: Data stored in epoch-based part directories
-4. **Mixed Data**: Each part contains data from **all measures** in the group
-5. **No Per-Measure Separation**: Cannot distinguish individual measures at file level
-
-## Proposed Design
-
-### 1. File Structure: `measure_migration_visitor.go`
-
-Create a new visitor implementing measure-specific file migration, following the exact pattern from `streamMigrationVisitor.VisitPart`:
-
-```go
-// measureMigrationVisitor implements file-based migration for measure data
-type measureMigrationVisitor struct {
- selector node.Selector // Node selector
- client queue.Client // Queue client
- chunkedClients map[string]queue.ChunkedSyncClient // Per-node chunked sync clients
- logger *logger.Logger
- progress *Progress // Progress tracker
- lfs fs.FileSystem
- group string
- targetShardNum uint32 // Target shard count
- replicas uint32 // Replica count
- chunkSize int // Chunk size for streaming
-}
-
-// Key methods based on streamMigrationVisitor pattern:
-func (mv *measureMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, seriesIndexPath string, shardIDs []common.ShardID) error // Not used for measures
-func (mv *measureMigrationVisitor) VisitPart(segmentTR *timestamp.TimeRange, sourceShardID common.ShardID, partPath string) error // Core migration logic
-```
-
-### 2. Integration Function: `measure_migration_integration.go`
-
-```go
-// migrateMeasureWithFileBasedAndProgress performs file-based measure migration
-// NOTE: measures parameter is removed since all measures in a group share the same parts
-func migrateMeasureWithFileBasedAndProgress(
- tsdbRootPath string,
- timeRange timestamp.TimeRange,
- group *commonv1.Group,
- nodeLabels map[string]string,
- nodes []*databasev1.Node,
- metadata metadata.Repo,
- logger *logger.Logger,
- progress *Progress,
- chunkSize int,
-) error
-```
-
-### 3. Core VisitPart Implementation
-
-The key difference from stream migration is that measures use the existing visitor pattern with `storage.VisitSegmentsInTimeRange` which calls `VisitPart` for each part. We adapt `streamMigrationVisitor.VisitPart` for measure data:
-
-#### VisitPart Method (adapted from streamMigrationVisitor.VisitPart:189-242)
-```go
-func (mv *measureMigrationVisitor) VisitPart(segmentTR *timestamp.TimeRange, sourceShardID common.ShardID, partPath string) error {
- // 1. Parse part ID from directory name (similar to stream.ParsePartMetadata but for measures)
- partID, err := mv.parsePartIDFromPath(partPath)
- if err != nil {
- return fmt.Errorf("failed to parse part ID from path %s: %w", partPath, err)
- }
-
- // 2. Check if this part has already been completed
- if mv.progress.IsMeasurePartCompleted(mv.group, partID) {
- mv.logger.Debug().Uint64("part_id", partID).Str("group", mv.group).Msg("part already completed, skipping")
- return nil
- }
-
- // 3. Calculate target shard ID based on source shard ID mapping
- targetShardID := mv.calculateTargetShardID(uint32(sourceShardID))
-
- // 4. Log migration start
- mv.logger.Info().
- Uint64("part_id", partID).
- Uint32("source_shard", uint32(sourceShardID)).
- Uint32("target_shard", targetShardID).
- Str("part_path", partPath).
- Str("group", mv.group).
- Msg("migrating measure part")
-
- // 5. Create file readers (adapted from stream.CreatePartFileReaderFromPath)
- files, release := mv.createMeasurePartFileReaders(partPath)
- defer release()
-
- // 6. Create streaming part data
- partData := mv.createStreamingPartData(partID, targetShardID, files, segmentTR)
-
- // 7. Stream entire part to target shard replicas
- if err := mv.streamPartToTargetShard(partData); err != nil {
- errorMsg := fmt.Sprintf("failed to stream part to target shard: %v", err)
- mv.progress.MarkMeasurePartError(mv.group, partID, errorMsg)
- return fmt.Errorf("failed to stream part to target shard: %w", err)
- }
-
- // 8. Mark part as completed in progress tracker
- mv.progress.MarkMeasurePartCompleted(mv.group, partID)
-
- // 9. Log completion
- mv.logger.Info().
- Uint64("part_id", partID).
- Str("group", mv.group).
- Int("completed_parts", mv.progress.GetMeasurePartProgress(mv.group)).
- Int("total_parts", mv.progress.GetMeasurePartCount(mv.group)).
- Msg("measure part migration completed successfully")
-
- return nil
-}
-```
-
-#### Supporting Methods
-```go
-// parsePartIDFromPath extracts part ID from the part directory name
-func (mv *measureMigrationVisitor) parsePartIDFromPath(partPath string) (uint64, error) {
- partDirName := filepath.Base(partPath)
- return strconv.ParseUint(partDirName, 16, 64)
-}
-
-// calculateTargetShardID maps source shard ID to target shard ID (same as stream)
-func (mv *measureMigrationVisitor) calculateTargetShardID(sourceShardID uint32) uint32 {
- return sourceShardID % mv.targetShardNum
-}
-
-// createMeasurePartFileReaders creates file readers for all files in a measure part
-// Based on measure/part.go structure: metadata.json, primary.bin, meta.bin, timestamps.bin, fv.bin, *.tf, *.tfm
-func (mv *measureMigrationVisitor) createMeasurePartFileReaders(partPath string) ([]queue.FileInfo, func()) {
- var files []queue.FileInfo
- var readers []fs.SequentialReader
-
- // Core measure files
- coreFiles := []string{"metadata.json", "primary.bin", "meta.bin", "timestamps.bin", "fv.bin"}
- for _, filename := range coreFiles {
- if reader := mv.openFileReader(partPath, filename); reader != nil {
- readers = append(readers, reader)
- files = append(files, queue.FileInfo{Name: filename, Reader: reader})
- }
- }
-
- // Tag family files (*.tf and *.tfm)
- entries := mv.lfs.ReadDir(partPath)
- for _, entry := range entries {
- if !entry.IsDir() && (strings.HasSuffix(entry.Name(), ".tf") || strings.HasSuffix(entry.Name(), ".tfm")) {
- if reader := mv.openFileReader(partPath, entry.Name()); reader != nil {
- readers = append(readers, reader)
- files = append(files, queue.FileInfo{Name: entry.Name(), Reader: reader})
- }
- }
- }
-
- // Return cleanup function
- release := func() {
- for _, reader := range readers {
- reader.Close()
- }
- }
-
- return files, release
-}
-```
-
-### 4. Integration with Existing Visitor Pattern
-
-The measure migration leverages the existing `storage.VisitSegmentsInTimeRange` infrastructure:
-
-#### Integration Function
-```go
-// migrateMeasureWithFileBasedAndProgress performs file-based measure migration
-func migrateMeasureWithFileBasedAndProgress(
- tsdbRootPath string,
- timeRange timestamp.TimeRange,
- group *commonv1.Group,
- nodeLabels map[string]string,
- nodes []*databasev1.Node,
- metadata metadata.Repo,
- logger *logger.Logger,
- progress *Progress,
- chunkSize int,
-) error {
- // Parse group configuration
- shardNum, replicas, ttl, selector, client, err := parseGroup(group, nodeLabels, nodes, logger, metadata)
- if err != nil {
- return err
- }
- defer client.GracefulStop()
-
- // Convert TTL to IntervalRule
- intervalRule := storage.MustToIntervalRule(ttl)
-
- // Create measure migration visitor
- visitor := newMeasureMigrationVisitor(group, shardNum, replicas, selector, client, logger, progress, chunkSize)
- defer visitor.Close()
-
- // Use existing segment visitor infrastructure
- return storage.VisitSegmentsInTimeRange(tsdbRootPath, timeRange, visitor, intervalRule)
-}
-```
-
-### 5. Service Integration
-
-The measure migration integrates seamlessly with the existing lifecycle service by replacing the query-based approach:
-
-#### Update `service.go`
-```go
-func (l *lifecycleService) processMeasureGroup(ctx context.Context, g *commonv1.Group, measureSVC measure.Service,
- nodes []*databasev1.Node, labels map[string]string, progress *Progress,
-) {
- // Check if file-based migration is enabled
- if l.useFileBasedMigration {
- l.processMeasureGroupFileBased(ctx, g, measureSVC, nodes, labels, progress)
- } else {
- l.processMeasureGroupQueryBased(ctx, g, measureSVC, nodes, labels, progress) // Keep existing logic
- }
-}
-
-func (l *lifecycleService) processMeasureGroupFileBased(ctx context.Context, g *commonv1.Group, measureSVC measure.Service,
- nodes []*databasev1.Node, labels map[string]string, progress *Progress,
-) {
- tr := measureSVC.GetRemovalSegmentsTimeRange(g.Metadata.Name)
- if tr.Start.IsZero() && tr.End.IsZero() {
- l.l.Info().Msgf("no removal segments time range for group %s, skipping measure migration", g.Metadata.Name)
- progress.MarkGroupCompleted(g.Metadata.Name)
- return
- }
-
- // Get measure snapshot directory
- measureDir := progress.SnapshotMeasureDir
- if measureDir == "" {
- l.l.Error().Msgf("no measure snapshot directory available for group: %s", g.Metadata.Name)
- return
- }
-
- // Perform file-based migration (no need for measure list since all measures share same parts)
- err = migrateMeasureWithFileBasedAndProgress(
- filepath.Join(measureDir, g.Metadata.Name),
- *tr,
- g,
- labels,
- nodes,
- l.metadata,
- l.l,
- progress,
- int(l.chunkSize),
- )
- if err != nil {
- l.l.Error().Err(err).Msgf("file-based measure migration failed for group: %s", g.Metadata.Name)
- return
- }
-
- l.l.Info().Msgf("completed file-based measure migration for group: %s", g.Metadata.Name)
- l.deleteExpiredMeasureSegments(ctx, g, tr, progress)
- progress.MarkGroupCompleted(g.Metadata.Name)
-}
-```
-
-### 6. Progress Tracking Enhancement
-
-#### Extend `progress.go`
-```go
-// Add measure part-specific progress tracking
-type Progress struct {
- // ... existing fields ...
-
- // Measure part progress tracking
- MeasurePartCounts map[string]int `json:"measure_part_counts"`
- MeasurePartProgress map[string]int `json:"measure_part_progress"`
- MeasurePartCompleted map[string]map[uint64]bool `json:"measure_part_completed"`
- MeasurePartErrors map[string]map[uint64]string `json:"measure_part_errors"`
-}
-
-func (p *Progress) IsMeasurePartCompleted(group string, partID uint64) bool
-func (p *Progress) MarkMeasurePartCompleted(group string, partID uint64)
-func (p *Progress) MarkMeasurePartError(group string, partID uint64, error string)
-func (p *Progress) SetMeasurePartCount(group string, totalParts int)
-func (p *Progress) GetMeasurePartProgress(group string) int
-```
-
-### 7. Queue Topic Extension
-
-#### Add New Topic in `api/data/topics.go`
-```go
-const (
- // ... existing topics ...
- TopicMeasurePartSync = Topic("measure-part-sync")
-)
-```
-
-### 8. Configuration Flag
-
-#### Add File-Based Migration Toggle
-```go
-func (l *lifecycleService) FlagSet() *run.FlagSet {
- flagS := run.NewFlagSet(l.Name())
- // ... existing flags ...
- flagS.BoolVar(&l.useFileBasedMigration, "file-based-migration", true,
- "Use file-based migration for measures (default: true, fallback to query-based)")
- return flagS
-}
-```
-
-## Implementation Benefits
-
-### Corrected Understanding Impact
-With the correct understanding that **all measures in a group share the same parts**, the file-based approach becomes even more beneficial:
-
-- **Massive efficiency gain**: Instead of querying each measure individually and reconstructing write requests, we stream the entire part containing **all measures at once**
-- **~100-1000x performance improvement** expected since we eliminate per-measure processing overhead entirely
-- **Single part = All measures**: Each part migration covers all measures in the group simultaneously
-- **Reduced complexity**: No need to track per-measure progress, only per-part progress
-- **Better resource utilization**: Stream complete data files rather than rebuilding data point by point
-
-### Performance Advantages
-1. **Bulk Transfer**: Stream entire part files containing all measures instead of individual records
-2. **Eliminated Per-Measure Processing**: No need to query and process each measure separately
-3. **Parallel Processing**: Multiple parts can be streamed concurrently
-4. **Atomic Group Operations**: Single part transfer covers entire group's measures
-
-### Reliability Improvements
-1. **Atomic Operations**: Either entire part (with all measures) succeeds or fails
-2. **Resume Capability**: Can restart from failed parts
-3. **Error Isolation**: Part-level error handling covers all measures
-4. **Consistent State**: No partial measure migrations within a part
-
-### Scalability Benefits
-1. **Memory Efficiency**: Stream files directly without data reconstruction
-2. **Network Optimization**: Bulk transfer vs thousands of individual messages
-3. **Resource Management**: Better control over concurrent transfers
-4. **Storage Efficiency**: Direct file-to-file transfers
-
-### Real-World Impact
-For a group with 10 measures and 1000 parts:
-- **Query-based**: 10,000 individual measure queries + 1,000,000 write requests
-- **File-based**: 1,000 part file transfers (covers all measures automatically)
-
-This represents a **1000x reduction** in operations and network round-trips.
-
-## Migration Strategy
-
-### Phase 1: Implementation
-1. Create `measure_migration_visitor.go`
-2. Create `measure_migration_integration.go`
-3. Extend progress tracking for measure parts
-4. Add new queue topic for measure part sync
-
-### Phase 2: Integration
-1. Update `service.go` with file-based measure processing
-2. Add configuration flag for migration method selection
-3. Implement fallback to query-based approach
-
-### Phase 3: Testing & Validation
-1. Unit tests for measure part discovery and streaming
-2. Integration tests comparing query-based vs file-based results
-3. Performance benchmarks and memory usage analysis
-4. End-to-end lifecycle migration testing
-
-### Phase 4: Rollout
-1. Default to file-based migration with query-based fallback
-2. Monitor performance improvements and error rates
-3. Deprecate query-based approach after validation period
-
-## Risk Mitigation
-
-### Compatibility
-- Maintain query-based fallback for compatibility
-- Validate file-based results match query-based results
-- Support rollback via configuration flag
-
-### Error Handling
-- Implement robust error recovery at part level
-- Maintain detailed error logging and progress tracking
-- Support manual intervention for failed parts
-
-### Performance
-- Implement streaming throttling to prevent resource exhaustion
-- Monitor memory usage during large part transfers
-- Add metrics for transfer rates and success rates
-
-This design provides a comprehensive approach to migrating measure data using the proven file-based streaming pattern while maintaining compatibility and reliability.
\ No newline at end of file
diff --git a/banyand/backup/lifecycle/file_migration_visitor.go b/banyand/backup/lifecycle/measure_migration_visitor.go
similarity index 52%
copy from banyand/backup/lifecycle/file_migration_visitor.go
copy to banyand/backup/lifecycle/measure_migration_visitor.go
index c5ea084e..0735b5ae 100644
--- a/banyand/backup/lifecycle/file_migration_visitor.go
+++ b/banyand/backup/lifecycle/measure_migration_visitor.go
@@ -27,16 +27,16 @@ import (
"github.com/apache/skywalking-banyandb/api/common"
"github.com/apache/skywalking-banyandb/api/data"
commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
+ "github.com/apache/skywalking-banyandb/banyand/measure"
"github.com/apache/skywalking-banyandb/banyand/queue"
- "github.com/apache/skywalking-banyandb/banyand/stream"
"github.com/apache/skywalking-banyandb/pkg/fs"
"github.com/apache/skywalking-banyandb/pkg/logger"
"github.com/apache/skywalking-banyandb/pkg/node"
"github.com/apache/skywalking-banyandb/pkg/timestamp"
)
-// streamMigrationVisitor implements the stream.Visitor interface for file-based migration.
-type streamMigrationVisitor struct {
+// measureMigrationVisitor implements the measure.Visitor interface for file-based migration.
+type measureMigrationVisitor struct {
selector node.Selector // From parseGroup - node selector
client queue.Client // From parseGroup - queue client
chunkedClients map[string]queue.ChunkedSyncClient // Per-node chunked sync clients cache
@@ -49,11 +49,11 @@ type streamMigrationVisitor struct {
chunkSize int // Chunk size for streaming data
}
-// newStreamMigrationVisitor creates a new file-based migration visitor.
-func newStreamMigrationVisitor(group *commonv1.Group, shardNum, replicas uint32, selector node.Selector, client queue.Client,
+// newMeasureMigrationVisitor creates a new file-based migration visitor.
+func newMeasureMigrationVisitor(group *commonv1.Group, shardNum, replicas uint32, selector node.Selector, client queue.Client,
l *logger.Logger, progress *Progress, chunkSize int,
-) *streamMigrationVisitor {
- return &streamMigrationVisitor{
+) *measureMigrationVisitor {
+ return &measureMigrationVisitor{
group: group.Metadata.Name,
targetShardNum: shardNum,
replicas: replicas,
@@ -67,14 +67,14 @@ func newStreamMigrationVisitor(group *commonv1.Group, shardNum, replicas uint32,
}
}
-// VisitSeries implements stream.Visitor.
-func (mv *streamMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, seriesIndexPath string, shardIDs []common.ShardID) error {
+// VisitSeries implements measure.Visitor.
+func (mv *measureMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, seriesIndexPath string, shardIDs []common.ShardID) error {
mv.logger.Info().
Str("path", seriesIndexPath).
Int64("min_timestamp", segmentTR.Start.UnixNano()).
Int64("max_timestamp", segmentTR.End.UnixNano()).
Str("group", mv.group).
- Msg("migrating series index")
+ Msg("migrating measure series index")
// Find all *.seg segment files in the seriesIndexPath
entries := mv.lfs.ReadDir(seriesIndexPath)
@@ -89,17 +89,17 @@ func (mv *streamMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, se
if len(segmentFiles) == 0 {
mv.logger.Debug().
Str("path", seriesIndexPath).
- Msg("no .seg files found in series index path")
+ Msg("no .seg files found in measure series index path")
return nil
}
mv.logger.Info().
Int("segment_count", len(segmentFiles)).
Str("path", seriesIndexPath).
- Msg("found segment files for migration")
+ Msg("found measure segment files for migration")
// Set the total number of series segments for progress tracking
- mv.SetStreamSeriesCount(len(segmentFiles))
+ mv.SetMeasureSeriesCount(len(segmentFiles))
// Create StreamingPartData for this segment
files := make([]queue.FileInfo, 0, len(segmentFiles))
@@ -123,12 +123,12 @@ func (mv *streamMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, se
shardID := common.ShardID(segmentID)
// Check if this segment has already been completed
- if mv.progress.IsStreamSeriesCompleted(mv.group, shardID) {
+ if mv.progress.IsMeasureSeriesCompleted(mv.group, shardID) {
mv.logger.Debug().
Uint64("segment_id", segmentID).
Str("filename", segmentFileName).
Str("group", mv.group).
- Msg("segment already completed, skipping")
+ Msg("measure segment already completed, skipping")
continue
}
@@ -136,19 +136,19 @@ func (mv *streamMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, se
Uint64("segment_id", segmentID).
Str("filename", segmentFileName).
Str("group", mv.group).
- Msg("migrating segment file")
+ Msg("migrating measure segment file")
// Create file reader for the segment file
segmentFilePath := filepath.Join(seriesIndexPath, segmentFileName)
segmentFile, err := mv.lfs.OpenFile(segmentFilePath)
if err != nil {
- errorMsg := fmt.Sprintf("failed to open segment file %s: %v", segmentFilePath, err)
- mv.progress.MarkStreamSeriesError(mv.group, shardID, errorMsg)
+ errorMsg := fmt.Sprintf("failed to open measure segment file %s: %v", segmentFilePath, err)
+ mv.progress.MarkMeasureSeriesError(mv.group, shardID, errorMsg)
mv.logger.Error().
Str("path", segmentFilePath).
Err(err).
- Msg("failed to open segment file")
- return fmt.Errorf("failed to open segment file %s: %w", segmentFilePath, err)
+ Msg("failed to open measure segment file")
+ return fmt.Errorf("failed to open measure segment file %s: %w", segmentFilePath, err)
}
// Close the file reader
@@ -163,43 +163,43 @@ func (mv *streamMigrationVisitor) VisitSeries(segmentTR *timestamp.TimeRange, se
Uint64("segment_id", segmentID).
Str("filename", segmentFileName).
Str("group", mv.group).
- Int("completed_segments", mv.progress.GetStreamSeriesProgress(mv.group)).
- Int("total_segments", mv.progress.GetStreamSeriesCount(mv.group)).
- Msg("segment migration completed successfully")
+ Int("completed_segments", mv.progress.GetMeasureSeriesProgress(mv.group)).
+ Int("total_segments", mv.progress.GetMeasureSeriesCount(mv.group)).
+ Msg("measure segment migration completed successfully")
}
// Send segment file to each shard in shardIDs
for _, shardID := range shardIDs {
targetShardID := mv.calculateTargetShardID(uint32(shardID))
- partData := mv.createStreamingSegmentFromFiles(targetShardID, files, segmentTR, data.TopicStreamSeriesSync.String())
+ partData := mv.createStreamingSegmentFromFiles(targetShardID, files, segmentTR, data.TopicMeasureSeriesIndexUpdate.String())
// Stream segment to target shard replicas
if err := mv.streamPartToTargetShard(partData); err != nil {
- errorMsg := fmt.Sprintf("failed to stream segment to target shard %d: %v", targetShardID, err)
- mv.progress.MarkStreamSeriesError(mv.group, shardID, errorMsg)
- return fmt.Errorf("failed to stream segment to target shard %d: %w", targetShardID, err)
+ errorMsg := fmt.Sprintf("failed to stream measure segment to target shard %d: %v", targetShardID, err)
+ mv.progress.MarkMeasureSeriesError(mv.group, shardID, errorMsg)
+ return fmt.Errorf("failed to stream measure segment to target shard %d: %w", targetShardID, err)
}
// Mark segment as completed
- mv.progress.MarkStreamSeriesCompleted(mv.group, shardID)
+ mv.progress.MarkMeasureSeriesCompleted(mv.group, shardID)
}
return nil
}
-// VisitPart implements stream.Visitor - core migration logic.
-func (mv *streamMigrationVisitor) VisitPart(_ *timestamp.TimeRange, sourceShardID common.ShardID, partPath string) error {
- partData, err := stream.ParsePartMetadata(mv.lfs, partPath)
+// VisitPart implements measure.Visitor - core migration logic.
+func (mv *measureMigrationVisitor) VisitPart(_ *timestamp.TimeRange, sourceShardID common.ShardID, partPath string) error {
+ partData, err := measure.ParsePartMetadata(mv.lfs, partPath)
if err != nil {
- return fmt.Errorf("failed to parse part metadata: %w", err)
+ return fmt.Errorf("failed to parse measure part metadata: %w", err)
}
partID := partData.ID
// Check if this part has already been completed
- if mv.progress.IsStreamPartCompleted(mv.group, partID) {
+ if mv.progress.IsMeasurePartCompleted(mv.group, partID) {
mv.logger.Debug().
Uint64("part_id", partID).
Str("group", mv.group).
- Msg("part already completed, skipping")
+ Msg("measure part already completed, skipping")
return nil
}
@@ -212,157 +212,43 @@ func (mv *streamMigrationVisitor) VisitPart(_ *timestamp.TimeRange, sourceShardI
Uint32("target_shard", targetShardID).
Str("part_path", partPath).
Str("group", mv.group).
- Msg("migrating part")
+ Msg("migrating measure part")
- files, release := stream.CreatePartFileReaderFromPath(partPath, mv.lfs)
+ files, release := measure.CreatePartFileReaderFromPath(partPath, mv.lfs)
defer release()
partData.Group = mv.group
partData.ShardID = targetShardID
- partData.Topic = data.TopicStreamPartSync.String()
+ partData.Topic = data.TopicMeasurePartSync.String()
partData.Files = files
// Stream entire part to target shard replicas
if err := mv.streamPartToTargetShard(partData); err != nil {
- errorMsg := fmt.Sprintf("failed to stream part to target shard: %v", err)
- mv.progress.MarkStreamPartError(mv.group, partID, errorMsg)
- return fmt.Errorf("failed to stream part to target shard: %w", err)
+ errorMsg := fmt.Sprintf("failed to stream measure part to target shard: %v", err)
+ mv.progress.MarkMeasurePartError(mv.group, partID, errorMsg)
+ return fmt.Errorf("failed to stream measure part to target shard: %w", err)
}
// Mark part as completed in progress tracker
- mv.progress.MarkStreamPartCompleted(mv.group, partID)
+ mv.progress.MarkMeasurePartCompleted(mv.group, partID)
mv.logger.Info().
Uint64("part_id", partID).
Str("group", mv.group).
- Int("completed_parts", mv.progress.GetStreamPartProgress(mv.group)).
- Int("total_parts", mv.progress.GetStreamPartCount(mv.group)).
- Msg("part migration completed successfully")
-
- return nil
-}
-
-// VisitElementIndex implements stream.Visitor.
-func (mv *streamMigrationVisitor) VisitElementIndex(segmentTR *timestamp.TimeRange, sourceShardID common.ShardID, indexPath string) error {
- if mv.progress.IsStreamElementIndexCompleted(mv.group) {
- mv.logger.Debug().
- Str("group", mv.group).
- Msg("element index segment already completed, skipping")
- return nil
- }
-
- mv.logger.Info().
- Str("path", indexPath).
- Uint32("shard_id", uint32(sourceShardID)).
- Int64("min_timestamp", segmentTR.Start.UnixNano()).
- Int64("max_timestamp", segmentTR.End.UnixNano()).
- Str("group", mv.group).
- Msg("migrating element index")
-
- // Find all .seg files in the element index directory
- entries := mv.lfs.ReadDir(indexPath)
-
- var segmentFiles []string
- for _, entry := range entries {
- if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".seg") {
- segmentFiles = append(segmentFiles, entry.Name())
- }
- }
-
- if len(segmentFiles) == 0 {
- mv.logger.Debug().
- Str("path", indexPath).
- Msg("no .seg files found in element index directory")
- return nil
- }
-
- // Set the total number of element index segment files for progress tracking
- mv.SetStreamElementIndexCount(len(segmentFiles))
-
- // Calculate target shard ID (using a simple approach for element index)
- targetShardID := mv.calculateTargetShardID(uint32(sourceShardID))
- mv.logger.Info().
- Int("segment_count", len(segmentFiles)).
- Str("path", indexPath).
- Uint32("source_shard", uint32(sourceShardID)).
- Uint32("target_shard", targetShardID).
- Msg("found element index segment files for migration")
-
- // Create FileInfo for this segment file
- files := make([]queue.FileInfo, 0, len(segmentFiles))
- // Process each segment file
- for _, segmentFileName := range segmentFiles {
- // Extract segment ID from filename (remove .seg extension)
- segmentIDStr := strings.TrimSuffix(segmentFileName, ".seg")
-
- // Parse hex segment ID
- segmentID, err := strconv.ParseUint(segmentIDStr, 16, 64)
- if err != nil {
- mv.logger.Error().
- Str("filename", segmentFileName).
- Str("id_str", segmentIDStr).
- Err(err).
- Msg("failed to parse segment ID from filename")
- continue
- }
-
- mv.logger.Info().
- Uint64("segment_id", segmentID).
- Str("filename", segmentFileName).
- Str("group", mv.group).
- Msg("migrating element index segment file")
-
- // Create file reader for the segment file
- segmentFilePath := filepath.Join(indexPath, segmentFileName)
- segmentFile, err := mv.lfs.OpenFile(segmentFilePath)
- if err != nil {
- mv.logger.Error().
- Str("path", segmentFilePath).
- Err(err).
- Msg("failed to open element index segment file")
- return fmt.Errorf("failed to open element index segment file %s: %w", segmentFilePath, err)
- }
-
- // Close the file reader
- defer segmentFile.Close()
-
- files = append(files, queue.FileInfo{
- Name: segmentFileName,
- Reader: segmentFile.SequentialRead(),
- })
-
- mv.logger.Info().
- Uint64("segment_id", segmentID).
- Str("filename", segmentFileName).
- Str("group", mv.group).
- Int("completed_segments", mv.progress.GetStreamElementIndexProgress(mv.group)).
- Int("total_segments", mv.progress.GetStreamElementIndexCount(mv.group)).
- Msg("element index segment migration completed successfully")
- }
- partData := mv.createStreamingSegmentFromFiles(targetShardID, files, segmentTR, data.TopicStreamElementIndexSync.String())
-
- // Stream segment file to target shard replicas
- if err := mv.streamPartToTargetShard(partData); err != nil {
- errorMsg := fmt.Sprintf("failed to stream element index to target shard: %v", err)
- mv.progress.MarkStreamElementIndexError(mv.group, errorMsg)
- return fmt.Errorf("failed to stream element index to target shard: %w", err)
- }
-
- // Mark segment as completed
- mv.progress.MarkStreamElementIndexCompleted(mv.group)
+ Int("completed_parts", mv.progress.GetMeasurePartProgress(mv.group)).
+ Int("total_parts", mv.progress.GetMeasurePartCount(mv.group)).
+ Msg("measure part migration completed successfully")
return nil
}
// calculateTargetShardID maps source shard ID to target shard ID.
-func (mv *streamMigrationVisitor) calculateTargetShardID(sourceShardID uint32) uint32 {
- // Simple modulo-based mapping from source shard to target shard
- // This ensures deterministic and balanced distribution
- return sourceShardID % mv.targetShardNum
+func (mv *measureMigrationVisitor) calculateTargetShardID(sourceShardID uint32) uint32 {
+ return calculateTargetShardID(sourceShardID, mv.targetShardNum)
}
// streamPartToTargetShard sends part data to all replicas of the target shard.
-func (mv *streamMigrationVisitor) streamPartToTargetShard(partData queue.StreamingPartData) error {
+func (mv *measureMigrationVisitor) streamPartToTargetShard(partData queue.StreamingPartData) error {
targetShardID := partData.ShardID
copies := mv.replicas + 1
@@ -376,7 +262,7 @@ func (mv *streamMigrationVisitor) streamPartToTargetShard(partData queue.Streami
// Stream part data to target node using chunked sync
if err := mv.streamPartToNode(nodeID, targetShardID, partData); err != nil {
- return fmt.Errorf("failed to stream part to node %s: %w", nodeID, err)
+ return fmt.Errorf("failed to stream measure part to node %s: %w", nodeID, err)
}
}
@@ -384,7 +270,7 @@ func (mv *streamMigrationVisitor) streamPartToTargetShard(partData queue.Streami
}
// streamPartToNode streams part data to a specific target node.
-func (mv *streamMigrationVisitor) streamPartToNode(nodeID string, targetShardID uint32, partData queue.StreamingPartData) error {
+func (mv *measureMigrationVisitor) streamPartToNode(nodeID string, targetShardID uint32, partData queue.StreamingPartData) error {
// Get or create chunked client for this node (cache hit optimization)
chunkedClient, exists := mv.chunkedClients[nodeID]
if !exists {
@@ -419,13 +305,13 @@ func (mv *streamMigrationVisitor) streamPartToNode(nodeID string, targetShardID
Uint32("target_shard", targetShardID).
Uint64("part_id", partData.ID).
Str("group", mv.group).
- Msg("file-based migration part completed successfully")
+ Msg("file-based measure migration part completed successfully")
return nil
}
// Close cleans up all chunked sync clients.
-func (mv *streamMigrationVisitor) Close() error {
+func (mv *measureMigrationVisitor) Close() error {
for nodeID, client := range mv.chunkedClients {
if err := client.Close(); err != nil {
mv.logger.Warn().Err(err).Str("node", nodeID).Msg("failed to close chunked sync client")
@@ -436,7 +322,7 @@ func (mv *streamMigrationVisitor) Close() error {
}
// createStreamingSegmentFromFiles creates StreamingPartData from segment files.
-func (mv *streamMigrationVisitor) createStreamingSegmentFromFiles(
+func (mv *measureMigrationVisitor) createStreamingSegmentFromFiles(
targetShardID uint32,
files []queue.FileInfo,
segmentTR *timestamp.TimeRange,
@@ -454,35 +340,24 @@ func (mv *streamMigrationVisitor) createStreamingSegmentFromFiles(
return segmentData
}
-// SetStreamPartCount sets the total number of parts for the current stream.
-func (mv *streamMigrationVisitor) SetStreamPartCount(totalParts int) {
+// SetMeasurePartCount sets the total number of parts for the current measure.
+func (mv *measureMigrationVisitor) SetMeasurePartCount(totalParts int) {
if mv.progress != nil {
- mv.progress.SetStreamPartCount(mv.group, totalParts)
+ mv.progress.SetMeasurePartCount(mv.group, totalParts)
mv.logger.Info().
Str("group", mv.group).
Int("total_parts", totalParts).
- Msg("set stream part count for progress tracking")
+ Msg("set measure part count for progress tracking")
}
}
-// SetStreamSeriesCount sets the total number of series segments for the current stream.
-func (mv *streamMigrationVisitor) SetStreamSeriesCount(totalSegments int) {
+// SetMeasureSeriesCount sets the total number of series segments for the current measure.
+func (mv *measureMigrationVisitor) SetMeasureSeriesCount(totalSegments int) {
if mv.progress != nil {
- mv.progress.SetStreamSeriesCount(mv.group, totalSegments)
+ mv.progress.SetMeasureSeriesCount(mv.group, totalSegments)
mv.logger.Info().
Str("group", mv.group).
Int("total_segments", totalSegments).
- Msg("set stream series count for progress tracking")
- }
-}
-
-// SetStreamElementIndexCount sets the total number of element index segment files for the current stream.
-func (mv *streamMigrationVisitor) SetStreamElementIndexCount(totalSegmentFiles int) {
- if mv.progress != nil {
- mv.progress.SetStreamElementIndexCount(mv.group, totalSegmentFiles)
- mv.logger.Info().
- Str("group", mv.group).
- Int("total_segment_files", totalSegmentFiles).
- Msg("set stream element index segment count for progress tracking")
+ Msg("set measure series count for progress tracking")
}
}
diff --git a/banyand/backup/lifecycle/file_migration_integration.go b/banyand/backup/lifecycle/migration_integration.go
similarity index 60%
rename from banyand/backup/lifecycle/file_migration_integration.go
rename to banyand/backup/lifecycle/migration_integration.go
index 3aacc949..be9aa164 100644
--- a/banyand/backup/lifecycle/file_migration_integration.go
+++ b/banyand/backup/lifecycle/migration_integration.go
@@ -22,6 +22,7 @@ import (
commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
"github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/banyand/measure"
"github.com/apache/skywalking-banyandb/banyand/metadata"
"github.com/apache/skywalking-banyandb/banyand/stream"
"github.com/apache/skywalking-banyandb/pkg/logger"
@@ -105,3 +106,76 @@ func (pcv *partCountVisitor) VisitPart(_ *timestamp.TimeRange, _ common.ShardID,
func (pcv *partCountVisitor) VisitElementIndex(_ *timestamp.TimeRange, _ common.ShardID, _ string) error {
return nil
}
+
+// migrateMeasureWithFileBasedAndProgress performs file-based measure migration with progress tracking.
+func migrateMeasureWithFileBasedAndProgress(
+ tsdbRootPath string,
+ timeRange timestamp.TimeRange,
+ group *commonv1.Group,
+ nodeLabels map[string]string,
+ nodes []*databasev1.Node,
+ metadata metadata.Repo,
+ logger *logger.Logger,
+ progress *Progress,
+ chunkSize int,
+) error {
+ // Use parseGroup function to get sharding parameters and TTL
+ shardNum, replicas, ttl, selector, client, err := parseGroup(group, nodeLabels, nodes, logger, metadata)
+ if err != nil {
+ return err
+ }
+ defer client.GracefulStop()
+
+ // Convert TTL to IntervalRule using storage.MustToIntervalRule
+ intervalRule := storage.MustToIntervalRule(ttl)
+
+ // Count total parts before starting migration
+ totalParts, err := countMeasureParts(tsdbRootPath, timeRange, intervalRule)
+ if err != nil {
+ logger.Warn().Err(err).Msg("failed to count measure parts, proceeding without part count")
+ } else {
+ logger.Info().Int("total_parts", totalParts).Msg("counted measure parts for progress tracking")
+ }
+
+ // Create file-based migration visitor with progress tracking
+ visitor := newMeasureMigrationVisitor(group, shardNum, replicas, selector, client, logger, progress, chunkSize)
+ defer visitor.Close()
+
+ // Set the total part count for progress tracking
+ if totalParts > 0 {
+ visitor.SetMeasurePartCount(totalParts)
+ }
+
+ // Use the existing VisitMeasuresInTimeRange function with our file-based visitor
+ return measure.VisitMeasuresInTimeRange(tsdbRootPath, timeRange, visitor, intervalRule)
+}
+
+// countMeasureParts counts the total number of parts in the given time range.
+func countMeasureParts(tsdbRootPath string, timeRange timestamp.TimeRange, intervalRule storage.IntervalRule) (int, error) {
+ // Create a simple visitor to count parts
+ partCounter := &measurePartCountVisitor{}
+
+ // Use the existing VisitMeasuresInTimeRange function to count parts
+ err := measure.VisitMeasuresInTimeRange(tsdbRootPath, timeRange, partCounter, intervalRule)
+ if err != nil {
+ return 0, err
+ }
+
+ return partCounter.partCount, nil
+}
+
+// measurePartCountVisitor is a simple visitor that counts measure parts.
+type measurePartCountVisitor struct {
+ partCount int
+}
+
+// VisitSeries implements measure.Visitor.
+func (pcv *measurePartCountVisitor) VisitSeries(_ *timestamp.TimeRange, _ string, _ []common.ShardID) error {
+ return nil
+}
+
+// VisitPart implements measure.Visitor.
+func (pcv *measurePartCountVisitor) VisitPart(_ *timestamp.TimeRange, _ common.ShardID, _ string) error {
+ pcv.partCount++
+ return nil
+}
diff --git a/banyand/backup/lifecycle/progress.go b/banyand/backup/lifecycle/progress.go
index 3b57bd6e..cb60e59d 100644
--- a/banyand/backup/lifecycle/progress.go
+++ b/banyand/backup/lifecycle/progress.go
@@ -31,11 +31,8 @@ import (
type Progress struct {
logger *logger.Logger `json:"-"`
CompletedGroups map[string]bool `json:"completed_groups"`
- CompletedMeasures map[string]map[string]bool `json:"completed_measures"`
DeletedStreamGroups map[string]bool `json:"deleted_stream_groups"`
DeletedMeasureGroups map[string]bool `json:"deleted_measure_groups"`
- MeasureErrors map[string]map[string]string `json:"measure_errors"`
- MeasureCounts map[string]map[string]int `json:"measure_counts"`
CompletedStreamParts map[string]map[uint64]bool `json:"completed_stream_parts"`
StreamPartErrors map[string]map[uint64]string `json:"stream_part_errors"`
CompletedStreamSeries map[string]map[common.ShardID]bool `json:"completed_stream_series"`
@@ -48,10 +45,19 @@ type Progress struct {
StreamSeriesProgress map[string]int `json:"stream_series_progress"`
StreamElementIndexCounts map[string]int `json:"stream_element_index_counts"`
StreamElementIndexProgress map[string]int `json:"stream_element_index_progress"`
- progressFilePath string `json:"-"`
- SnapshotStreamDir string `json:"snapshot_stream_dir"`
- SnapshotMeasureDir string `json:"snapshot_measure_dir"`
- mu sync.Mutex `json:"-"`
+ // Measure part-specific progress tracking
+ CompletedMeasureParts map[string]map[uint64]bool `json:"completed_measure_parts"`
+ MeasurePartErrors map[string]map[uint64]string `json:"measure_part_errors"`
+ MeasurePartCounts map[string]int `json:"measure_part_counts"`
+ MeasurePartProgress map[string]int `json:"measure_part_progress"`
+ CompletedMeasureSeries map[string]map[common.ShardID]bool `json:"completed_measure_series"`
+ MeasureSeriesErrors map[string]map[common.ShardID]string `json:"measure_series_errors"`
+ MeasureSeriesCounts map[string]int `json:"measure_series_counts"`
+ MeasureSeriesProgress map[string]int `json:"measure_series_progress"`
+ progressFilePath string `json:"-"`
+ SnapshotStreamDir string `json:"snapshot_stream_dir"`
+ SnapshotMeasureDir string `json:"snapshot_measure_dir"`
+ mu sync.Mutex `json:"-"`
}
// AllGroupsFullyCompleted checks if all groups are fully completed.
@@ -71,11 +77,8 @@ func (p *Progress) AllGroupsFullyCompleted(groups []*commonv1.Group) bool {
func NewProgress(path string, l *logger.Logger) *Progress {
return &Progress{
CompletedGroups: make(map[string]bool),
- CompletedMeasures: make(map[string]map[string]bool),
DeletedStreamGroups: make(map[string]bool),
DeletedMeasureGroups: make(map[string]bool),
- MeasureErrors: make(map[string]map[string]string),
- MeasureCounts: make(map[string]map[string]int),
CompletedStreamParts: make(map[string]map[uint64]bool),
StreamPartErrors: make(map[string]map[uint64]string),
StreamPartCounts: make(map[string]int),
@@ -88,6 +91,14 @@ func NewProgress(path string, l *logger.Logger) *Progress {
StreamElementIndexErrors: make(map[string]string),
StreamElementIndexCounts: make(map[string]int),
StreamElementIndexProgress: make(map[string]int),
+ CompletedMeasureParts: make(map[string]map[uint64]bool),
+ MeasurePartErrors: make(map[string]map[uint64]string),
+ MeasurePartCounts: make(map[string]int),
+ MeasurePartProgress: make(map[string]int),
+ CompletedMeasureSeries: make(map[string]map[common.ShardID]bool),
+ MeasureSeriesErrors: make(map[string]map[common.ShardID]string),
+ MeasureSeriesCounts: make(map[string]int),
+ MeasureSeriesProgress: make(map[string]int),
progressFilePath: path,
logger: l,
}
@@ -170,31 +181,6 @@ func (p *Progress) IsGroupCompleted(group string) bool {
return p.CompletedGroups[group]
}
-// MarkMeasureCompleted marks a measure as completed.
-func (p *Progress) MarkMeasureCompleted(group, measure string, count int) {
- defer p.saveProgress()
- p.mu.Lock()
- defer p.mu.Unlock()
- if p.CompletedMeasures[group] == nil {
- p.CompletedMeasures[group] = make(map[string]bool)
- }
- p.CompletedMeasures[group][measure] = true
- if p.MeasureCounts[group] == nil {
- p.MeasureCounts[group] = make(map[string]int)
- }
- p.MeasureCounts[group][measure] = count
-}
-
-// IsMeasureCompleted checks if a measure has been completed.
-func (p *Progress) IsMeasureCompleted(group, measure string) bool {
- p.mu.Lock()
- defer p.mu.Unlock()
- if measures, ok := p.CompletedMeasures[group]; ok {
- return measures[measure]
- }
- return false
-}
-
// MarkStreamGroupDeleted marks a stream group segments as deleted.
func (p *Progress) MarkStreamGroupDeleted(group string) {
defer p.saveProgress()
@@ -225,28 +211,6 @@ func (p *Progress) IsMeasureGroupDeleted(group string) bool {
return p.DeletedMeasureGroups[group]
}
-// ClearErrors resets all prior measure error records.
-func (p *Progress) ClearErrors() {
- p.mu.Lock()
- defer p.mu.Unlock()
- p.MeasureErrors = make(map[string]map[string]string)
-}
-
-// MarkMeasureError records an error message for a specific measure.
-func (p *Progress) MarkMeasureError(group, measure, msg string) {
- defer p.saveProgress()
- p.mu.Lock()
- defer p.mu.Unlock()
- if p.MeasureErrors[group] == nil {
- p.MeasureErrors[group] = make(map[string]string)
- }
- p.MeasureErrors[group][measure] = msg
- if p.CompletedMeasures[group] == nil {
- p.CompletedMeasures[group] = make(map[string]bool)
- }
- p.CompletedMeasures[group][measure] = false
-}
-
// Remove deletes the progress file.
func (p *Progress) Remove(path string, l *logger.Logger) {
if path == "" {
@@ -277,9 +241,6 @@ func (p *Progress) MarkStreamPartCompleted(group string, partID uint64) {
p.CompletedStreamParts[group][partID] = true
// Update progress count
- if p.StreamPartProgress[group] == 0 {
- p.StreamPartProgress[group] = 0
- }
p.StreamPartProgress[group]++
}
@@ -315,15 +276,7 @@ func (p *Progress) SetStreamPartCount(group string, totalParts int) {
p.mu.Lock()
defer p.mu.Unlock()
- if p.StreamPartCounts[group] == 0 {
- p.StreamPartCounts[group] = 0
- }
p.StreamPartCounts[group] = totalParts
-
- // Initialize progress tracking
- if p.StreamPartProgress[group] == 0 {
- p.StreamPartProgress[group] = 0
- }
}
// GetStreamPartCount returns the total number of parts for a stream.
@@ -348,41 +301,6 @@ func (p *Progress) GetStreamPartProgress(group string) int {
return 0
}
-// IsStreamFullyCompleted checks if all parts of a stream have been completed.
-func (p *Progress) IsStreamFullyCompleted(group string) bool {
- p.mu.Lock()
- defer p.mu.Unlock()
-
- totalParts := p.StreamPartCounts[group]
- completedParts := p.StreamPartProgress[group]
-
- return totalParts > 0 && completedParts >= totalParts
-}
-
-// GetStreamPartErrors returns all errors for a specific stream.
-func (p *Progress) GetStreamPartErrors(group string) map[uint64]string {
- p.mu.Lock()
- defer p.mu.Unlock()
-
- if errors, ok := p.StreamPartErrors[group]; ok {
- // Return a copy to avoid concurrent access issues
- result := make(map[uint64]string)
- for k, v := range errors {
- result[k] = v
- }
- return result
- }
- return nil
-}
-
-// ClearStreamPartErrors clears all errors for a specific stream.
-func (p *Progress) ClearStreamPartErrors(group string) {
- p.mu.Lock()
- defer p.mu.Unlock()
-
- delete(p.StreamPartErrors, group)
-}
-
// MarkStreamSeriesCompleted marks a specific series segment of a stream as completed.
func (p *Progress) MarkStreamSeriesCompleted(group string, shardID common.ShardID) {
defer p.saveProgress()
@@ -398,9 +316,6 @@ func (p *Progress) MarkStreamSeriesCompleted(group string, shardID common.ShardI
p.CompletedStreamSeries[group][shardID] = true
// Update progress count
- if p.StreamSeriesProgress[group] == 0 {
- p.StreamSeriesProgress[group] = 0
- }
p.StreamSeriesProgress[group]++
}
@@ -436,15 +351,7 @@ func (p *Progress) SetStreamSeriesCount(group string, totalSegments int) {
p.mu.Lock()
defer p.mu.Unlock()
- if p.StreamSeriesCounts[group] == 0 {
- p.StreamSeriesCounts[group] = 0
- }
p.StreamSeriesCounts[group] = totalSegments
-
- // Initialize progress tracking
- if p.StreamSeriesProgress[group] == 0 {
- p.StreamSeriesProgress[group] = 0
- }
}
// GetStreamSeriesCount returns the total number of series segments for a stream.
@@ -469,17 +376,6 @@ func (p *Progress) GetStreamSeriesProgress(group string) int {
return 0
}
-// IsStreamSeriesFullyCompleted checks if all series segments of a stream have been completed.
-func (p *Progress) IsStreamSeriesFullyCompleted(group string) bool {
- p.mu.Lock()
- defer p.mu.Unlock()
-
- totalSegments := p.StreamSeriesCounts[group]
- completedSegments := p.StreamSeriesProgress[group]
-
- return totalSegments > 0 && completedSegments >= totalSegments
-}
-
// GetStreamSeriesErrors returns all errors for a specific stream series.
func (p *Progress) GetStreamSeriesErrors(group string) map[common.ShardID]string {
p.mu.Lock()
@@ -514,9 +410,6 @@ func (p *Progress) MarkStreamElementIndexCompleted(group string) {
p.CompletedStreamElementIndex[group] = true
// Update progress count
- if p.StreamElementIndexProgress[group] == 0 {
- p.StreamElementIndexProgress[group] = 0
- }
p.StreamElementIndexProgress[group]++
}
@@ -544,15 +437,7 @@ func (p *Progress) SetStreamElementIndexCount(group string, totalIndexFiles int)
p.mu.Lock()
defer p.mu.Unlock()
- if p.StreamElementIndexCounts[group] == 0 {
- p.StreamElementIndexCounts[group] = 0
- }
p.StreamElementIndexCounts[group] = totalIndexFiles
-
- // Initialize progress tracking
- if p.StreamElementIndexProgress[group] == 0 {
- p.StreamElementIndexProgress[group] = 0
- }
}
// GetStreamElementIndexCount returns the total number of element index files for a stream.
@@ -577,35 +462,163 @@ func (p *Progress) GetStreamElementIndexProgress(group string) int {
return 0
}
-// IsStreamElementIndexFullyCompleted checks if all element index files of a stream have been completed.
-func (p *Progress) IsStreamElementIndexFullyCompleted(group string) bool {
+// MarkMeasurePartCompleted marks a specific part of a measure as completed.
+func (p *Progress) MarkMeasurePartCompleted(group string, partID uint64) {
+ defer p.saveProgress()
p.mu.Lock()
defer p.mu.Unlock()
- totalIndexFiles := p.StreamElementIndexCounts[group]
- completedIndexFiles := p.StreamElementIndexProgress[group]
+ // Initialize nested maps if they don't exist
+ if p.CompletedMeasureParts[group] == nil {
+ p.CompletedMeasureParts[group] = make(map[uint64]bool)
+ }
+
+ // Mark part as completed
+ p.CompletedMeasureParts[group][partID] = true
- return totalIndexFiles > 0 && completedIndexFiles >= totalIndexFiles
+ // Update progress count
+ p.MeasurePartProgress[group]++
}
-// GetStreamElementIndexErrors returns all errors for a specific stream element index.
-func (p *Progress) GetStreamElementIndexErrors(group string) map[uint64]string {
+// IsMeasurePartCompleted checks if a specific part of a measure has been completed.
+func (p *Progress) IsMeasurePartCompleted(group string, partID uint64) bool {
p.mu.Lock()
defer p.mu.Unlock()
- if errorMsg, ok := p.StreamElementIndexErrors[group]; ok && errorMsg != "" {
- // Return a map with a single entry since we now store errors at group level
- result := make(map[uint64]string)
- result[0] = errorMsg // Use 0 as a placeholder since we no longer track individual indexFileIDs
- return result
+ if parts, ok := p.CompletedMeasureParts[group]; ok {
+ return parts[partID]
}
- return nil
+ return false
+}
+
+// MarkMeasurePartError records an error for a specific part of a measure.
+func (p *Progress) MarkMeasurePartError(group string, partID uint64, errorMsg string) {
+ defer p.saveProgress()
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ // Initialize nested maps if they don't exist
+ if p.MeasurePartErrors[group] == nil {
+ p.MeasurePartErrors[group] = make(map[uint64]string)
+ }
+
+ // Record the error
+ p.MeasurePartErrors[group][partID] = errorMsg
+}
+
+// SetMeasurePartCount sets the total number of parts for a measure.
+func (p *Progress) SetMeasurePartCount(group string, totalParts int) {
+ defer p.saveProgress()
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ p.MeasurePartCounts[group] = totalParts
+}
+
+// GetMeasurePartCount returns the total number of parts for a measure.
+func (p *Progress) GetMeasurePartCount(group string) int {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ if counts, ok := p.MeasurePartCounts[group]; ok {
+ return counts
+ }
+ return 0
+}
+
+// GetMeasurePartProgress returns the number of completed parts for a measure.
+func (p *Progress) GetMeasurePartProgress(group string) int {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ if progress, ok := p.MeasurePartProgress[group]; ok {
+ return progress
+ }
+ return 0
+}
+
+// ClearErrors clears all errors for a specific group.
+func (p *Progress) ClearErrors() {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ p.StreamPartErrors = make(map[string]map[uint64]string)
+ p.StreamSeriesErrors = make(map[string]map[common.ShardID]string)
+ p.StreamElementIndexErrors = make(map[string]string)
+ p.MeasurePartErrors = make(map[string]map[uint64]string)
+ p.MeasureSeriesErrors = make(map[string]map[common.ShardID]string)
+}
+
+// MarkMeasureSeriesCompleted marks a specific series segment of a measure as completed.
+func (p *Progress) MarkMeasureSeriesCompleted(group string, shardID common.ShardID) {
+ defer p.saveProgress()
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ // Initialize nested maps if they don't exist
+ if p.CompletedMeasureSeries[group] == nil {
+ p.CompletedMeasureSeries[group] = make(map[common.ShardID]bool)
+ }
+
+ // Mark series segment as completed
+ p.CompletedMeasureSeries[group][shardID] = true
+
+ // Update progress count
+ p.MeasureSeriesProgress[group]++
+}
+
+// IsMeasureSeriesCompleted checks if a specific series segment of a measure has been completed.
+func (p *Progress) IsMeasureSeriesCompleted(group string, shardID common.ShardID) bool {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ if segments, ok := p.CompletedMeasureSeries[group]; ok {
+ return segments[shardID]
+ }
+ return false
+}
+
+// MarkMeasureSeriesError records an error for a specific series segment of a measure.
+func (p *Progress) MarkMeasureSeriesError(group string, shardID common.ShardID, errorMsg string) {
+ defer p.saveProgress()
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ // Initialize nested maps if they don't exist
+ if p.MeasureSeriesErrors[group] == nil {
+ p.MeasureSeriesErrors[group] = make(map[common.ShardID]string)
+ }
+
+ // Record the error
+ p.MeasureSeriesErrors[group][shardID] = errorMsg
+}
+
+// SetMeasureSeriesCount sets the total number of series segments for a measure.
+func (p *Progress) SetMeasureSeriesCount(group string, totalSegments int) {
+ defer p.saveProgress()
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ p.MeasureSeriesCounts[group] = totalSegments
+}
+
+// GetMeasureSeriesCount returns the total number of series segments for a measure.
+func (p *Progress) GetMeasureSeriesCount(group string) int {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ if counts, ok := p.MeasureSeriesCounts[group]; ok {
+ return counts
+ }
+ return 0
}
-// ClearStreamElementIndexErrors clears all errors for a specific stream element index.
-func (p *Progress) ClearStreamElementIndexErrors(group string) {
+// GetMeasureSeriesProgress returns the number of completed series segments for a measure.
+func (p *Progress) GetMeasureSeriesProgress(group string) int {
p.mu.Lock()
defer p.mu.Unlock()
- delete(p.StreamElementIndexErrors, group)
+ if progress, ok := p.MeasureSeriesProgress[group]; ok {
+ return progress
+ }
+ return 0
}
diff --git a/banyand/backup/lifecycle/progress_test.go b/banyand/backup/lifecycle/progress_test.go
deleted file mode 100644
index 15fa3dd1..00000000
--- a/banyand/backup/lifecycle/progress_test.go
+++ /dev/null
@@ -1,145 +0,0 @@
-// Licensed to Apache Software Foundation (ASF) under one or more contributor
-// license agreements. See the NOTICE file distributed with
-// this work for additional information regarding copyright
-// ownership. Apache Software Foundation (ASF) licenses this file to you under
-// the Apache License, Version 2.0 (the "License"); you may
-// not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package lifecycle
-
-import (
- "os"
- "path/filepath"
- "testing"
-
- "github.com/stretchr/testify/assert"
- "github.com/stretchr/testify/require"
-
- "github.com/apache/skywalking-banyandb/pkg/logger"
-)
-
-func TestProgress(t *testing.T) {
- tmpDir, err := os.MkdirTemp("", "progress-test")
- require.NoError(t, err)
- defer os.RemoveAll(tmpDir)
-
- progressPath := filepath.Join(tmpDir, "progress.json")
- l := logger.GetLogger("test")
-
- t.Run("NewProgress", func(t *testing.T) {
- progress := NewProgress(progressPath, l)
- assert.NotNil(t, progress)
- assert.Empty(t, progress.CompletedGroups)
- assert.Empty(t, progress.CompletedMeasures)
- assert.Empty(t, progress.DeletedStreamGroups)
- assert.Empty(t, progress.DeletedMeasureGroups)
- })
-
- t.Run("MarkAndCheckItems", func(t *testing.T) {
- progress := NewProgress(progressPath, l)
-
- progress.MarkGroupCompleted("group1")
- assert.True(t, progress.IsGroupCompleted("group1"))
- assert.False(t, progress.IsGroupCompleted("group2"))
-
- progress.MarkMeasureCompleted("group1", "measure1", 1)
- assert.True(t, progress.IsMeasureCompleted("group1", "measure1"))
- assert.False(t, progress.IsMeasureCompleted("group1", "measure2"))
- assert.False(t, progress.IsMeasureCompleted("group2", "measure1"))
-
- progress.MarkStreamGroupDeleted("group1")
- assert.True(t, progress.IsStreamGroupDeleted("group1"))
- assert.False(t, progress.IsStreamGroupDeleted("group2"))
-
- progress.MarkMeasureGroupDeleted("group1")
- assert.True(t, progress.IsMeasureGroupDeleted("group1"))
- assert.False(t, progress.IsMeasureGroupDeleted("group2"))
- })
-
- t.Run("SaveAndLoad", func(t *testing.T) {
- progress := NewProgress(progressPath, l)
- progress.MarkGroupCompleted("group1")
- progress.MarkMeasureCompleted("group1", "measure1", 1)
- progress.MarkStreamGroupDeleted("group2")
- progress.MarkMeasureGroupDeleted("group2")
-
- progress.Save(progressPath, l)
-
- loaded := LoadProgress(progressPath, l)
-
- assert.True(t, loaded.IsGroupCompleted("group1"))
- assert.True(t, loaded.IsMeasureCompleted("group1", "measure1"))
- assert.True(t, loaded.IsStreamGroupDeleted("group2"))
- assert.True(t, loaded.IsMeasureGroupDeleted("group2"))
-
- assert.False(t, loaded.IsGroupCompleted("group3"))
- assert.False(t, loaded.IsMeasureCompleted("group1", "measure2"))
- assert.False(t, loaded.IsStreamGroupDeleted("group3"))
- assert.False(t, loaded.IsMeasureGroupDeleted("group3"))
- })
-
- t.Run("LoadNonExistent", func(t *testing.T) {
- nonExistentPath := filepath.Join(tmpDir, "nonexistent.json")
- progress := LoadProgress(nonExistentPath, l)
- assert.NotNil(t, progress)
- assert.Empty(t, progress.CompletedGroups)
- })
-
- t.Run("RemoveProgressFile", func(t *testing.T) {
- progress := NewProgress(progressPath, l)
- progress.MarkGroupCompleted("group1")
- progress.Save(progressPath, l)
-
- _, err := os.Stat(progressPath)
- assert.NoError(t, err)
-
- progress.Remove(progressPath, l)
-
- _, err = os.Stat(progressPath)
- assert.True(t, os.IsNotExist(err))
- })
-
- t.Run("StreamSeriesProgress", func(t *testing.T) {
- progress := NewProgress(progressPath, l)
-
- // Test series progress tracking
- progress.SetStreamSeriesCount("group1", 5)
- assert.Equal(t, 5, progress.GetStreamSeriesCount("group1"))
- assert.Equal(t, 0, progress.GetStreamSeriesProgress("group1"))
-
- // Mark some series segments as completed
- progress.MarkStreamSeriesCompleted("group1", 1)
- progress.MarkStreamSeriesCompleted("group1", 2)
- assert.Equal(t, 2, progress.GetStreamSeriesProgress("group1"))
- assert.True(t, progress.IsStreamSeriesCompleted("group1", 1))
- assert.True(t, progress.IsStreamSeriesCompleted("group1", 2))
- assert.False(t, progress.IsStreamSeriesCompleted("group1", 3))
-
- // Test error tracking
- progress.MarkStreamSeriesError("group1", 3, "test error")
- errors := progress.GetStreamSeriesErrors("group1")
- assert.Equal(t, "test error", errors[3])
-
- // Test completion check
- assert.False(t, progress.IsStreamSeriesFullyCompleted("group1"))
- progress.MarkStreamSeriesCompleted("group1", 3)
- progress.MarkStreamSeriesCompleted("group1", 4)
- progress.MarkStreamSeriesCompleted("group1", 5)
- assert.True(t, progress.IsStreamSeriesFullyCompleted("group1"))
-
- // Test error clearing
- progress.ClearStreamSeriesErrors("group1")
- errors = progress.GetStreamSeriesErrors("group1")
- assert.Nil(t, errors)
- })
-}
diff --git a/banyand/backup/lifecycle/service.go b/banyand/backup/lifecycle/service.go
index 90a5ccd0..88dac8dd 100644
--- a/banyand/backup/lifecycle/service.go
+++ b/banyand/backup/lifecycle/service.go
@@ -39,16 +39,10 @@ import (
streamv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/stream/v1"
"github.com/apache/skywalking-banyandb/banyand/backup/snapshot"
"github.com/apache/skywalking-banyandb/banyand/internal/storage"
- "github.com/apache/skywalking-banyandb/banyand/measure"
"github.com/apache/skywalking-banyandb/banyand/metadata"
- "github.com/apache/skywalking-banyandb/banyand/metadata/schema"
"github.com/apache/skywalking-banyandb/banyand/observability"
"github.com/apache/skywalking-banyandb/banyand/protector"
- "github.com/apache/skywalking-banyandb/banyand/queue"
"github.com/apache/skywalking-banyandb/pkg/logger"
- "github.com/apache/skywalking-banyandb/pkg/node"
- pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
- "github.com/apache/skywalking-banyandb/pkg/query/model"
"github.com/apache/skywalking-banyandb/pkg/run"
"github.com/apache/skywalking-banyandb/pkg/timestamp"
)
@@ -196,14 +190,6 @@ func (l *lifecycleService) action() error {
Str("measure_snapshot", measureDir).
Msg("created snapshots")
progress.Save(l.progressFilePath, l.l)
- measureSVC, err := l.setupQuerySvc(ctx, measureDir)
- if measureSVC != nil {
- defer measureSVC.GracefulStop()
- }
- if err != nil {
- l.l.Error().Err(err).Msg("failed to setup query service")
- return err
- }
nodes, err := l.metadata.NodeRegistry().ListNode(ctx, databasev1.Role_ROLE_DATA)
if err != nil {
@@ -223,13 +209,12 @@ func (l *lifecycleService) action() error {
}
l.processStreamGroup(ctx, g, streamDir, nodes, labels, progress)
case commonv1.Catalog_CATALOG_MEASURE:
- if measureSVC == nil {
- l.l.Error().Msgf("measure service is not available, skipping group: %s", g.Metadata.Name)
- progress.MarkMeasureError(g.Metadata.Name, "", fmt.Sprintf("measure service unavailable for group %s", g.Metadata.Name))
- allGroupsCompleted = false
+ if measureDir == "" {
+ l.l.Warn().Msgf("measure snapshot directory is not available, skipping group: %s", g.Metadata.Name)
+ progress.MarkGroupCompleted(g.Metadata.Name)
continue
}
- l.processMeasureGroup(ctx, g, measureSVC, nodes, labels, progress)
+ l.processMeasureGroup(ctx, g, measureDir, nodes, labels, progress)
default:
l.l.Info().Msgf("group catalog: %s doesn't support lifecycle management", g.Catalog)
}
@@ -247,51 +232,21 @@ func (l *lifecycleService) action() error {
return fmt.Errorf("lifecycle migration partially completed, progress file retained; %d groups not fully completed", len(groups)-len(progress.CompletedGroups))
}
-// generateReport gathers counts & errors from Progress, writes one JSON file per run, and keeps only 5 latest.
+// generateReport gathers detailed counts & errors from Progress, writes comprehensive JSON file per run, and keeps only 5 latest.
func (l *lifecycleService) generateReport(p *Progress) {
- type grp struct {
- MeasureCounts map[string]int `json:"measure_counts"`
- MeasureErrors map[string]string `json:"measure_errors"`
- Name string `json:"name"`
- }
reportDir := l.reportDir
if err := os.MkdirAll(reportDir, 0o755); err != nil {
l.l.Error().Err(err).Msgf("failed to create report dir %s", reportDir)
return
}
- // build report
- var groups []grp
- for name := range p.CompletedGroups {
- groups = append(groups, grp{
- Name: name,
- MeasureCounts: p.MeasureCounts[name],
- MeasureErrors: p.MeasureErrors[name],
- })
- }
- // also include groups that had only measures
- for name := range p.CompletedMeasures {
- if _, seen := p.CompletedGroups[name]; seen {
- continue
- }
- groups = append(groups, grp{
- Name: name,
- MeasureCounts: p.MeasureCounts[name],
- MeasureErrors: p.MeasureErrors[name],
- })
- }
- payload := struct {
- GeneratedAt time.Time `json:"generated_at"`
- Groups []grp `json:"groups"`
- }{
- GeneratedAt: time.Now(),
- Groups: groups,
- }
+ // Build comprehensive migration report
+ report := l.buildMigrationReport(p)
// write file
fname := time.Now().Format("20060102_150405") + ".json"
fpath := filepath.Join(reportDir, fname)
- data, err := json.MarshalIndent(payload, "", " ")
+ data, err := json.MarshalIndent(report, "", " ")
if err != nil {
l.l.Error().Err(err).Msg("failed to marshal migration report")
return
@@ -300,18 +255,179 @@ func (l *lifecycleService) generateReport(p *Progress) {
l.l.Error().Err(err).Msgf("failed to write report file %s", fpath)
return
}
- l.l.Info().Msgf("wrote migration report %s", fpath)
+ l.l.Info().Msgf("wrote comprehensive migration report %s", fpath)
// rotate: keep only 5 latest
+ l.rotateReportFiles(reportDir)
+}
+
+// buildMigrationReport creates a comprehensive migration report from progress data.
+func (l *lifecycleService) buildMigrationReport(p *Progress) map[string]interface{} {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+
+ now := time.Now()
+ report := map[string]interface{}{
+ "generated_at": now,
+ "report_version": "2.0",
+ "summary": l.buildSummaryStats(p),
+ "errors": l.buildErrorSummary(p),
+ "snapshot_info": map[string]interface{}{
+ "stream_dir": p.SnapshotStreamDir,
+ "measure_dir": p.SnapshotMeasureDir,
+ },
+ }
+
+ return report
+}
+
+// buildSummaryStats creates overall migration statistics.
+func (l *lifecycleService) buildSummaryStats(p *Progress) map[string]interface{} {
+ totalGroups := len(p.CompletedGroups) + len(p.DeletedStreamGroups) + len(p.DeletedMeasureGroups)
+ completedGroups := len(p.CompletedGroups)
+
+ // Calculate total parts and series across all groups
+ totalStreamParts, completedStreamParts := l.calculateTotalCounts(p.StreamPartCounts, p.StreamPartProgress)
+ totalStreamSeries, completedStreamSeries := l.calculateTotalCounts(p.StreamSeriesCounts, p.StreamSeriesProgress)
+ totalStreamElementIndex, completedStreamElementIndex := l.calculateTotalCounts(p.StreamElementIndexCounts, p.StreamElementIndexProgress)
+ totalMeasureParts, completedMeasureParts := l.calculateTotalCounts(p.MeasurePartCounts, p.MeasurePartProgress)
+ totalMeasureSeries, completedMeasureSeries := l.calculateTotalCounts(p.MeasureSeriesCounts, p.MeasureSeriesProgress)
+
+ // Calculate error counts
+ streamPartErrors := l.countErrors(p.StreamPartErrors)
+ streamSeriesErrors := l.countErrors(p.StreamSeriesErrors)
+ streamElementIndexErrors := len(p.StreamElementIndexErrors)
+ measurePartErrors := l.countErrors(p.MeasurePartErrors)
+ measureSeriesErrors := l.countErrors(p.MeasureSeriesErrors)
+
+ return map[string]interface{}{
+ "migration_status": map[string]interface{}{
+ "total_groups": totalGroups,
+ "completed_groups": completedGroups,
+ "completion_rate": l.calculatePercentage(completedGroups, totalGroups),
+ },
+ "stream_migration": map[string]interface{}{
+ "parts": map[string]interface{}{
+ "total": totalStreamParts,
+ "completed": completedStreamParts,
+ "errors": streamPartErrors,
+ "completion_rate": l.calculatePercentage(completedStreamParts, totalStreamParts),
+ },
+ "series": map[string]interface{}{
+ "total": totalStreamSeries,
+ "completed": completedStreamSeries,
+ "errors": streamSeriesErrors,
+ "completion_rate": l.calculatePercentage(completedStreamSeries, totalStreamSeries),
+ },
+ "element_index": map[string]interface{}{
+ "total": totalStreamElementIndex,
+ "completed": completedStreamElementIndex,
+ "errors": streamElementIndexErrors,
+ "completion_rate": l.calculatePercentage(completedStreamElementIndex, totalStreamElementIndex),
+ },
+ },
+ "measure_migration": map[string]interface{}{
+ "parts": map[string]interface{}{
+ "total": totalMeasureParts,
+ "completed": completedMeasureParts,
+ "errors": measurePartErrors,
+ "completion_rate": l.calculatePercentage(completedMeasureParts, totalMeasureParts),
+ },
+ "series": map[string]interface{}{
+ "total": totalMeasureSeries,
+ "completed": completedMeasureSeries,
+ "errors": measureSeriesErrors,
+ "completion_rate": l.calculatePercentage(completedMeasureSeries, totalMeasureSeries),
+ },
+ },
+ }
+}
+
+// buildErrorSummary creates detailed error information.
+func (l *lifecycleService) buildErrorSummary(p *Progress) map[string]interface{} {
+ errors := map[string]interface{}{
+ "stream_parts": l.buildErrorDetails(p.StreamPartErrors),
+ "stream_series": l.buildErrorDetails(p.StreamSeriesErrors),
+ "stream_element_index": p.StreamElementIndexErrors,
+ "measure_parts": l.buildErrorDetails(p.MeasurePartErrors),
+ "measure_series": l.buildErrorDetails(p.MeasureSeriesErrors),
+ }
+
+ return errors
+}
+
+// Helper functions.
+func (l *lifecycleService) calculateTotalCounts(counts, progress map[string]int) (int, int) {
+ totalCount, totalProgress := 0, 0
+ for _, count := range counts {
+ totalCount += count
+ }
+ for _, prog := range progress {
+ totalProgress += prog
+ }
+ return totalCount, totalProgress
+}
+
+func (l *lifecycleService) countErrors(errorMaps interface{}) int {
+ switch v := errorMaps.(type) {
+ case map[string]map[uint64]string:
+ total := 0
+ for _, groupErrors := range v {
+ total += len(groupErrors)
+ }
+ return total
+ case map[string]map[string]string:
+ total := 0
+ for _, groupErrors := range v {
+ total += len(groupErrors)
+ }
+ return total
+ default:
+ return 0
+ }
+}
+
+func (l *lifecycleService) calculatePercentage(completed, total int) float64 {
+ if total == 0 {
+ return 0.0
+ }
+ return float64(completed) / float64(total) * 100.0
+}
+
+func (l *lifecycleService) buildErrorDetails(errorMaps interface{}) map[string]interface{} {
+ result := make(map[string]interface{})
+
+ switch v := errorMaps.(type) {
+ case map[string]map[uint64]string:
+ for group, groupErrors := range v {
+ if len(groupErrors) > 0 {
+ result[group] = groupErrors
+ }
+ }
+ case map[string]map[string]string:
+ for group, groupErrors := range v {
+ if len(groupErrors) > 0 {
+ result[group] = groupErrors
+ }
+ }
+ }
+
+ return result
+}
+
+// rotateReportFiles keeps only the 5 most recent report files.
+func (l *lifecycleService) rotateReportFiles(reportDir string) {
entries, err := os.ReadDir(reportDir)
if err != nil {
return
}
- type fi struct {
+
+ type fileInfo struct {
t time.Time
name string
}
- var list []fi
+
+ var list []fileInfo
for _, e := range entries {
if e.IsDir() {
continue
@@ -320,15 +436,20 @@ func (l *lifecycleService) generateReport(p *Progress) {
if err != nil {
continue
}
- list = append(list, fi{name: e.Name(), t: info.ModTime()})
+ list = append(list, fileInfo{name: e.Name(), t: info.ModTime()})
}
+
sort.Slice(list, func(i, j int) bool {
return list[i].t.After(list[j].t)
})
+
for i := 5; i < len(list); i++ {
_ = os.Remove(filepath.Join(reportDir, list[i].name))
}
- l.l.Info().Msg("rotated old migration reports")
+
+ if len(list) > 5 {
+ l.l.Info().Msgf("rotated %d old migration reports", len(list)-5)
+ }
}
func getGroupNames(groups []*commonv1.Group) []string {
@@ -487,23 +608,10 @@ func (l *lifecycleService) deleteExpiredStreamSegments(ctx context.Context, g *c
progress.Save(l.progressFilePath, l.l)
}
-func (l *lifecycleService) processMeasureGroup(ctx context.Context, g *commonv1.Group, measureSVC measure.Service,
+func (l *lifecycleService) processMeasureGroup(ctx context.Context, g *commonv1.Group, measureDir string,
nodes []*databasev1.Node, labels map[string]string, progress *Progress,
) {
- shardNum, replicas, _, selector, client, err := parseGroup(g, labels, nodes, l.l, l.metadata)
- if err != nil {
- l.l.Error().Err(err).Msgf("failed to parse group %s", g.Metadata.Name)
- return
- }
- defer client.GracefulStop()
-
- mm, err := l.metadata.MeasureRegistry().ListMeasure(ctx, schema.ListOpt{Group: g.Metadata.Name})
- if err != nil {
- l.l.Error().Err(err).Msgf("failed to list measures in group %s", g.Metadata.Name)
- return
- }
-
- tr := measureSVC.GetRemovalSegmentsTimeRange(g.Metadata.Name)
+ tr := l.getRemovalSegmentsTimeRange(g)
if tr.Start.IsZero() && tr.End.IsZero() {
l.l.Info().Msgf("no removal segments time range for group %s, skipping measure migration", g.Metadata.Name)
progress.MarkGroupCompleted(g.Metadata.Name)
@@ -511,98 +619,47 @@ func (l *lifecycleService) processMeasureGroup(ctx context.Context, g *commonv1.
return
}
- l.processMeasures(ctx, g, mm, measureSVC, tr, shardNum, replicas, selector, client, progress)
-
- allMeasuresDone := true
- for _, m := range mm {
- if !progress.IsMeasureCompleted(g.Metadata.Name, m.Metadata.Name) {
- allMeasuresDone = false
- break
- }
- }
- if allMeasuresDone {
- l.l.Info().Msgf("deleting expired measure segments for group: %s", g.Metadata.Name)
- l.deleteExpiredMeasureSegments(ctx, g, tr, progress)
- progress.MarkGroupCompleted(g.Metadata.Name)
- progress.Save(l.progressFilePath, l.l)
- } else {
- l.l.Info().Msgf("skipping delete expired measure segments for group %s: some measures not fully migrated", g.Metadata.Name)
+ // Try file-based migration first
+ if err := l.processMeasureGroupFileBased(ctx, g, measureDir, tr, nodes, labels, progress); err != nil {
+ l.l.Error().Err(err).Msgf("failed to migrate measure group %s using file-based approach", g.Metadata.Name)
+ return
}
-}
-func (l *lifecycleService) processMeasures(
- ctx context.Context,
- g *commonv1.Group,
- measures []*databasev1.Measure,
- measureSVC measure.Service,
- tr *timestamp.TimeRange,
- shardNum uint32,
- replicas uint32,
- selector node.Selector,
- client queue.Client,
- progress *Progress,
-) {
- for _, m := range measures {
- if progress.IsMeasureCompleted(g.Metadata.Name, m.Metadata.Name) {
- l.l.Debug().Msgf("skipping already completed measure: %s/%s", g.Metadata.Name, m.Metadata.Name)
- continue
- }
- sum, err := l.processSingleMeasure(ctx, m, measureSVC, tr, shardNum, replicas, selector, client)
- if err != nil {
- progress.MarkMeasureError(g.Metadata.Name, m.Metadata.Name, err.Error())
- } else {
- if sum < 1 {
- l.l.Debug().Msgf("no elements migrated for measure %s", m.Metadata.Name)
- } else {
- l.l.Info().Msgf("migrated %d elements in measure %s", sum, m.Metadata.Name)
- }
- progress.MarkMeasureCompleted(g.Metadata.Name, m.Metadata.Name, sum)
- }
- progress.Save(l.progressFilePath, l.l)
- }
+ l.l.Info().Msgf("deleting expired measure segments for group: %s", g.Metadata.Name)
+ l.deleteExpiredMeasureSegments(ctx, g, tr, progress)
+ progress.MarkGroupCompleted(g.Metadata.Name)
+ progress.Save(l.progressFilePath, l.l)
}
-func (l *lifecycleService) processSingleMeasure(ctx context.Context, m *databasev1.Measure,
- measureSVC measure.Service, tr *timestamp.TimeRange, shardNum uint32, replicas uint32, selector node.Selector, client queue.Client,
-) (int, error) {
- q, err := measureSVC.Measure(m.Metadata)
- if err != nil {
- l.l.Error().Err(err).Msgf("failed to get measure %s", m.Metadata.Name)
- return 0, err
+// processMeasureGroupFileBased uses file-based migration instead of query-based migration.
+func (l *lifecycleService) processMeasureGroupFileBased(_ context.Context, g *commonv1.Group,
+ measureDir string, tr *timestamp.TimeRange, nodes []*databasev1.Node, labels map[string]string, progress *Progress,
+) error {
+ if progress.IsMeasureGroupDeleted(g.Metadata.Name) {
+ l.l.Info().Msgf("skipping already completed file-based measure migration for group: %s", g.Metadata.Name)
+ return nil
}
- tagProjection := make([]model.TagProjection, len(m.TagFamilies))
- for i, tf := range m.TagFamilies {
- tagProjection[i] = model.TagProjection{
- Family: tf.Name,
- Names: make([]string, len(tf.Tags)),
- }
- for j, t := range tf.Tags {
- tagProjection[i].Names[j] = t.Name
- }
- }
- fieldProjection := make([]string, len(m.Fields))
- for i, f := range m.Fields {
- fieldProjection[i] = f.Name
- }
- entity := make([]*modelv1.TagValue, len(m.Entity.TagNames))
- for idx := range m.Entity.TagNames {
- entity[idx] = pbv1.AnyTagValue
- }
+ l.l.Info().Msgf("starting file-based measure migration for group: %s", g.Metadata.Name)
- result, err := q.Query(ctx, model.MeasureQueryOptions{
- Name: m.Metadata.Name,
- TagProjection: tagProjection,
- FieldProjection: fieldProjection,
- Entities: [][]*modelv1.TagValue{entity},
- TimeRange: tr,
- })
+ // Use the file-based migration with existing visitor pattern
+ err := migrateMeasureWithFileBasedAndProgress(
+ filepath.Join(measureDir, g.Metadata.Name), // Use snapshot directory as source
+ *tr, // Time range for segments to migrate
+ g, // Group configuration
+ labels, // Node labels
+ nodes, // Target nodes
+ l.metadata, // Metadata repository
+ l.l, // Logger
+ progress, // Progress tracking
+ int(l.chunkSize), // Chunk size for streaming
+ )
if err != nil {
- l.l.Error().Err(err).Msgf("failed to query measure %s", m.Metadata.Name)
- return 0, err
+ return fmt.Errorf("file-based measure migration failed: %w", err)
}
- return migrateMeasure(ctx, m, result, shardNum, replicas, selector, client, l.l), nil
+ l.l.Info().Msgf("completed file-based measure migration for group: %s", g.Metadata.Name)
+ return nil
}
func (l *lifecycleService) deleteExpiredMeasureSegments(ctx context.Context, g *commonv1.Group, tr *timestamp.TimeRange, progress *Progress) {
diff --git a/banyand/backup/lifecycle/steps.go b/banyand/backup/lifecycle/steps.go
index ce76a222..72387962 100644
--- a/banyand/backup/lifecycle/steps.go
+++ b/banyand/backup/lifecycle/steps.go
@@ -18,34 +18,21 @@
package lifecycle
import (
- "context"
- "encoding/base64"
"fmt"
- "time"
"github.com/pkg/errors"
- "google.golang.org/protobuf/types/known/timestamppb"
- "github.com/apache/skywalking-banyandb/api/common"
"github.com/apache/skywalking-banyandb/api/data"
commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
- measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
- modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
- streamv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/stream/v1"
"github.com/apache/skywalking-banyandb/banyand/backup/snapshot"
"github.com/apache/skywalking-banyandb/banyand/liaison/grpc"
- "github.com/apache/skywalking-banyandb/banyand/measure"
"github.com/apache/skywalking-banyandb/banyand/metadata"
"github.com/apache/skywalking-banyandb/banyand/metadata/schema"
"github.com/apache/skywalking-banyandb/banyand/queue"
"github.com/apache/skywalking-banyandb/banyand/queue/pub"
- "github.com/apache/skywalking-banyandb/pkg/bus"
- "github.com/apache/skywalking-banyandb/pkg/convert"
"github.com/apache/skywalking-banyandb/pkg/logger"
"github.com/apache/skywalking-banyandb/pkg/node"
- "github.com/apache/skywalking-banyandb/pkg/partition"
- "github.com/apache/skywalking-banyandb/pkg/query/model"
)
func (l *lifecycleService) getSnapshots(groups []*commonv1.Group, p *Progress) (streamDir string, measureDir string, err error) {
@@ -84,23 +71,6 @@ func (l *lifecycleService) getSnapshots(groups []*commonv1.Group, p *Progress) (
return streamDir, measureDir, nil
}
-func (l *lifecycleService) setupQuerySvc(ctx context.Context, measureDir string) (measureSVC measure.Service, err error) {
- ctx = context.WithValue(ctx, common.ContextNodeKey, common.Node{})
- if measureDir != "" {
- measureSVC, err = measure.NewReadonlyDataSVC(l.metadata, l.omr, l.pm)
- if err != nil {
- return nil, err
- }
- if err = measureSVC.FlagSet().Parse([]string{"--measure-data-path", measureDir}); err != nil {
- return nil, err
- }
- if err = measureSVC.PreRun(ctx); err != nil {
- return nil, err
- }
- }
- return measureSVC, nil
-}
-
func parseGroup(g *commonv1.Group, nodeLabels map[string]string, nodes []*databasev1.Node,
l *logger.Logger, metadata metadata.Repo,
) (uint32, uint32, *commonv1.IntervalRule, node.Selector, queue.Client, error) {
@@ -166,131 +136,3 @@ func parseGroup(g *commonv1.Group, nodeLabels map[string]string, nodes []*databa
}
return nst.ShardNum, nst.Replicas, nst.Ttl, nodeSel, client, nil
}
-
-func migrateStream(ctx context.Context, s *databasev1.Stream, result model.StreamQueryResult,
- shardNum uint32, replicas uint32, selector node.Selector, client queue.Client, l *logger.Logger,
-) (sum int) {
- if result == nil {
- return 0
- }
- defer result.Release()
- copies := replicas + 1
-
- entityLocator := partition.NewEntityLocator(s.TagFamilies, s.Entity, 0)
-
- batch := client.NewBatchPublisher(30 * time.Second)
- defer batch.Close()
- for sr := result.Pull(ctx); sr != nil; sr = result.Pull(ctx) {
- for i := range sr.ElementIDs {
- writeEntity := &streamv1.WriteRequest{
- Metadata: s.Metadata,
- Element: &streamv1.ElementValue{},
- }
- ev := writeEntity.Element
- ev.ElementId = base64.StdEncoding.EncodeToString(convert.Uint64ToBytes(sr.ElementIDs[i]))
- ev.Timestamp = timestamppb.New(time.Unix(0, sr.Timestamps[i]))
- for _, tf := range sr.TagFamilies {
- tfw := &modelv1.TagFamilyForWrite{}
- for _, tag := range tf.Tags {
- tfw.Tags = append(tfw.Tags, tag.Values[i])
- }
- ev.TagFamilies = append(ev.TagFamilies, tfw)
- }
- tagValues, shardID, err := entityLocator.Locate(s.Metadata.Name, ev.TagFamilies, shardNum)
- if err != nil {
- l.Error().Err(err).Msg("failed to locate entity")
- continue
- }
-
- // Write to multiple replicas
- for replicaID := uint32(0); replicaID < copies; replicaID++ {
- nodeID, err := selector.Pick(s.Metadata.Group, s.Metadata.Name, uint32(shardID), replicaID)
- if err != nil {
- l.Error().Err(err).Msg("failed to pick node")
- continue
- }
- iwr := &streamv1.InternalWriteRequest{
- Request: writeEntity,
- ShardId: uint32(shardID),
- EntityValues: tagValues[1:].Encode(),
- }
- message := bus.NewBatchMessageWithNode(bus.MessageID(time.Now().UnixNano()), nodeID, iwr)
- _, err = batch.Publish(ctx, data.TopicStreamWrite, message)
- if err != nil {
- l.Error().Err(err).Msg("failed to publish message")
- continue
- }
- }
- sum++
- }
- }
- return sum
-}
-
-func migrateMeasure(ctx context.Context, m *databasev1.Measure, result model.MeasureQueryResult,
- shardNum uint32, replicas uint32, selector node.Selector, client queue.Client, l *logger.Logger,
-) (sum int) {
- if result == nil {
- return 0
- }
- defer result.Release()
- copies := replicas + 1
-
- entityLocator := partition.NewEntityLocator(m.TagFamilies, m.Entity, 0)
-
- batch := client.NewBatchPublisher(30 * time.Second)
- defer batch.Close()
- for mr := result.Pull(); mr != nil; mr = result.Pull() {
- for i := range mr.Timestamps {
- writeRequest := &measurev1.WriteRequest{
- Metadata: m.Metadata,
- DataPoint: &measurev1.DataPointValue{
- Timestamp: timestamppb.New(time.Unix(0, mr.Timestamps[i])),
- Version: mr.Versions[i],
- },
- MessageId: uint64(time.Now().UnixNano()),
- }
-
- for _, tf := range mr.TagFamilies {
- tfWrite := &modelv1.TagFamilyForWrite{}
- for _, tag := range tf.Tags {
- tfWrite.Tags = append(tfWrite.Tags, tag.Values[i])
- }
- writeRequest.DataPoint.TagFamilies = append(writeRequest.DataPoint.TagFamilies, tfWrite)
- }
-
- for _, field := range mr.Fields {
- writeRequest.DataPoint.Fields = append(writeRequest.DataPoint.Fields, field.Values[i])
- }
-
- tagValues, shardID, err := entityLocator.Locate(m.Metadata.Name, writeRequest.DataPoint.TagFamilies, shardNum)
- if err != nil {
- l.Error().Err(err).Msg("failed to locate entity")
- continue
- }
-
- // Write to multiple replicas
- for replicaID := uint32(0); replicaID < copies; replicaID++ {
- nodeID, err := selector.Pick(m.Metadata.Group, m.Metadata.Name, uint32(shardID), replicaID)
- if err != nil {
- l.Error().Err(err).Msg("failed to pick node")
- continue
- }
-
- iwr := &measurev1.InternalWriteRequest{
- Request: writeRequest,
- ShardId: uint32(shardID),
- EntityValues: tagValues[1:].Encode(),
- }
-
- message := bus.NewBatchMessageWithNode(bus.MessageID(time.Now().UnixNano()), nodeID, iwr)
- _, err = batch.Publish(ctx, data.TopicMeasureWrite, message)
- if err != nil {
- l.Error().Err(err).Msg("failed to publish message")
- }
- }
- sum++
- }
- }
- return sum
-}
diff --git a/banyand/backup/lifecycle/steps_test.go b/banyand/backup/lifecycle/steps_test.go
deleted file mode 100644
index 2b53214c..00000000
--- a/banyand/backup/lifecycle/steps_test.go
+++ /dev/null
@@ -1,625 +0,0 @@
-// Licensed to Apache Software Foundation (ASF) under one or more contributor
-// license agreements. See the NOTICE file distributed with
-// this work for additional information regarding copyright
-// ownership. Apache Software Foundation (ASF) licenses this file to you under
-// the Apache License, Version 2.0 (the "License"); you may
-// not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package lifecycle
-
-import (
- "context"
- "encoding/base64"
- "testing"
- "time"
-
- "github.com/stretchr/testify/assert"
- "github.com/stretchr/testify/require"
- "go.uber.org/mock/gomock"
-
- "github.com/apache/skywalking-banyandb/api/common"
- "github.com/apache/skywalking-banyandb/api/data"
- commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
- databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
- measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
- modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
- streamv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/stream/v1"
- "github.com/apache/skywalking-banyandb/banyand/metadata"
- "github.com/apache/skywalking-banyandb/banyand/metadata/schema"
- "github.com/apache/skywalking-banyandb/banyand/queue"
- "github.com/apache/skywalking-banyandb/pkg/bus"
- "github.com/apache/skywalking-banyandb/pkg/convert"
- "github.com/apache/skywalking-banyandb/pkg/logger"
- mock_node "github.com/apache/skywalking-banyandb/pkg/node/mock"
- "github.com/apache/skywalking-banyandb/pkg/query/model"
-)
-
-// MockStreamQueryResult is a mock implementation of model.StreamQueryResult.
-type MockStreamQueryResult struct {
- results []*model.StreamResult
- index int
-}
-
-func (m *MockStreamQueryResult) Pull(_ context.Context) *model.StreamResult {
- if m.index >= len(m.results) {
- return nil
- }
- result := m.results[m.index]
- m.index++
- return result
-}
-
-func (m *MockStreamQueryResult) Release() {}
-
-// MockMeasureQueryResult is a mock implementation of model.MeasureQueryResult.
-type MockMeasureQueryResult struct {
- results []*model.MeasureResult
- index int
-}
-
-func (m *MockMeasureQueryResult) Pull() *model.MeasureResult {
- if m.index >= len(m.results) {
- return nil
- }
- result := m.results[m.index]
- m.index++
- return result
-}
-
-func (m *MockMeasureQueryResult) Release() {}
-
-func TestMigrateStream(t *testing.T) {
- ctrl := gomock.NewController(t)
- defer ctrl.Finish()
-
- mockClient := queue.NewMockClient(ctrl)
- mockBatchPublisher := queue.NewMockBatchPublisher(ctrl)
- mockSelector := mock_node.NewMockSelector(ctrl)
-
- l := logger.GetLogger("test")
-
- ctx := context.Background()
- stream := &databasev1.Stream{
- Metadata: &commonv1.Metadata{
- Name: "test-stream",
- Group: "test-group",
- },
- Entity: &databasev1.Entity{
- TagNames: []string{"service_id", "service_instance_id"},
- },
- TagFamilies: []*databasev1.TagFamilySpec{
- {
- Name: "default",
- Tags: []*databasev1.TagSpec{
- {
- Name: "service_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- {
- Name: "service_instance_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- {
- Name: "trace_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- },
- },
- },
- }
-
- shardNum := uint32(2)
- replicas := uint32(0)
-
- // Create stream result data with expected timestamps
- timestamp1 := int64(1672531200000000000) // First timestamp in nanoseconds
- timestamp2 := int64(1672531260000000000) // Second timestamp in nanoseconds
- elementID1 := uint64(1001)
- elementID2 := uint64(1002)
-
- streamResult := &model.StreamResult{
- Timestamps: []int64{timestamp1, timestamp2},
- ElementIDs: []uint64{elementID1, elementID2},
- TagFamilies: []model.TagFamily{
- {
- Name: "default",
- Tags: []model.Tag{
- {
- Name: "service_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-2"}}},
- },
- },
- {
- Name: "service_instance_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-2"}}},
- },
- },
- {
- Name: "trace_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "trace-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "trace-2"}}},
- },
- },
- },
- },
- },
- SIDs: []common.SeriesID{101, 102},
- }
-
- queryResult := &MockStreamQueryResult{
- results: []*model.StreamResult{streamResult},
- }
-
- mockClient.EXPECT().NewBatchPublisher(gomock.Any()).Return(mockBatchPublisher)
- mockBatchPublisher.EXPECT().Close().Return(nil, nil)
-
- mockSelector.EXPECT().Pick(stream.Metadata.Group, stream.Metadata.Name, gomock.Any(), gomock.Any()).Return("node-1", nil).Times(2)
-
- // Expected element IDs encoded as base64 strings
- expectedElementID1 := base64.StdEncoding.EncodeToString(convert.Uint64ToBytes(elementID1))
- expectedElementID2 := base64.StdEncoding.EncodeToString(convert.Uint64ToBytes(elementID2))
-
- // Use a counter to check the correct element for each call
- callCount := 0
- mockBatchPublisher.EXPECT().
- Publish(gomock.Any(), data.TopicStreamWrite, gomock.Any()).
- DoAndReturn(func(_ context.Context, _ bus.Topic, msg bus.Message) (string, error) {
- assert.Equal(t, "node-1", msg.Node())
-
- req, ok := msg.Data().(*streamv1.InternalWriteRequest)
- assert.True(t, ok)
- assert.NotNil(t, req.Request)
- assert.Equal(t, stream.Metadata, req.Request.Metadata)
-
- element := req.Request.Element
- assert.NotNil(t, element)
-
- if callCount == 0 {
- assert.Equal(t, expectedElementID1, element.ElementId)
- expectedTime := time.Unix(0, timestamp1)
- assert.Equal(t, expectedTime.UnixNano(), element.Timestamp.AsTime().UnixNano())
- } else {
- assert.Equal(t, expectedElementID2, element.ElementId)
- expectedTime := time.Unix(0, timestamp2)
- assert.Equal(t, expectedTime.UnixNano(), element.Timestamp.AsTime().UnixNano())
- }
-
- assert.Equal(t, 1, len(element.TagFamilies))
-
- tagFamily := element.TagFamilies[0]
- assert.Equal(t, 3, len(tagFamily.Tags))
-
- if callCount == 0 {
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-1"}}, tagFamily.Tags[0].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-1"}}, tagFamily.Tags[1].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "trace-1"}}, tagFamily.Tags[2].Value)
- } else {
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-2"}}, tagFamily.Tags[0].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-2"}}, tagFamily.Tags[1].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "trace-2"}}, tagFamily.Tags[2].Value)
- }
-
- callCount++
- return "", nil
- }).Times(2)
-
- migrateStream(ctx, stream, queryResult, shardNum, replicas, mockSelector, mockClient, l)
-
- assert.Equal(t, 1, queryResult.index)
- assert.Equal(t, 2, callCount, "Expected exactly 2 elements to be processed")
-}
-
-func TestMigrateMeasure(t *testing.T) {
- ctrl := gomock.NewController(t)
- defer ctrl.Finish()
-
- mockClient := queue.NewMockClient(ctrl)
- mockBatchPublisher := queue.NewMockBatchPublisher(ctrl)
- mockSelector := mock_node.NewMockSelector(ctrl)
-
- l := logger.GetLogger("test")
-
- ctx := context.Background()
- measure := &databasev1.Measure{
- Metadata: &commonv1.Metadata{
- Name: "test-measure",
- Group: "test-group",
- },
- Entity: &databasev1.Entity{
- TagNames: []string{"service_id", "service_instance_id"},
- },
- TagFamilies: []*databasev1.TagFamilySpec{
- {
- Name: "default",
- Tags: []*databasev1.TagSpec{
- {
- Name: "service_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- {
- Name: "service_instance_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- {
- Name: "endpoint_id",
- Type: databasev1.TagType_TAG_TYPE_STRING,
- },
- },
- },
- },
- Fields: []*databasev1.FieldSpec{
- {
- Name: "latency",
-
- FieldType: databasev1.FieldType_FIELD_TYPE_INT,
- },
- {
- Name: "status_code",
- FieldType: databasev1.FieldType_FIELD_TYPE_INT,
- },
- },
- }
-
- shardNum := uint32(2)
- replicas := uint32(0)
-
- // Create measure result data with expected timestamps
- timestamp1 := int64(1672531200000000000) // First timestamp in nanoseconds
- timestamp2 := int64(1672531260000000000) // Second timestamp in nanoseconds
-
- measureResult := &model.MeasureResult{
- Timestamps: []int64{timestamp1, timestamp2},
- Versions: []int64{101, 102},
- TagFamilies: []model.TagFamily{
- {
- Name: "default",
- Tags: []model.Tag{
- {
- Name: "service_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-2"}}},
- },
- },
- {
- Name: "service_instance_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-2"}}},
- },
- },
- {
- Name: "endpoint_id",
- Values: []*modelv1.TagValue{
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "endpoint-1"}}},
- {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "endpoint-2"}}},
- },
- },
- },
- },
- },
- Fields: []model.Field{
- {
- Name: "latency",
- Values: []*modelv1.FieldValue{
- {Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 100}}},
- {Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 200}}},
- },
- },
- {
- Name: "status_code",
- Values: []*modelv1.FieldValue{
- {Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 200}}},
- {Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 500}}},
- },
- },
- },
- SID: 101,
- }
-
- queryResult := &MockMeasureQueryResult{
- results: []*model.MeasureResult{measureResult},
- }
-
- mockClient.EXPECT().NewBatchPublisher(gomock.Any()).Return(mockBatchPublisher)
- mockBatchPublisher.EXPECT().Close().Return(nil, nil)
-
- mockSelector.EXPECT().Pick(measure.Metadata.Group, measure.Metadata.Name, gomock.Any(), gomock.Any()).
- Return("node-1", nil).Times(2)
-
- // Use a counter to check the correct element for each call
- callCount := 0
- mockBatchPublisher.EXPECT().
- Publish(gomock.Any(), data.TopicMeasureWrite, gomock.Any()).
- DoAndReturn(func(_ context.Context, _ bus.Topic, msg bus.Message) (string, error) {
- assert.Equal(t, "node-1", msg.Node())
-
- req, ok := msg.Data().(*measurev1.InternalWriteRequest)
- assert.True(t, ok)
- assert.NotNil(t, req.Request)
- assert.Equal(t, measure.Metadata, req.Request.Metadata)
-
- dataPoint := req.Request.DataPoint
- assert.NotNil(t, dataPoint)
-
- if callCount == 0 {
- expectedTime := time.Unix(0, timestamp1)
- assert.Equal(t, expectedTime.UnixNano(), dataPoint.Timestamp.AsTime().UnixNano())
- } else {
- expectedTime := time.Unix(0, timestamp2)
- assert.Equal(t, expectedTime.UnixNano(), dataPoint.Timestamp.AsTime().UnixNano())
- }
-
- assert.Equal(t, 1, len(dataPoint.TagFamilies))
- tagFamily := dataPoint.TagFamilies[0]
- assert.Equal(t, 3, len(tagFamily.Tags))
-
- if callCount == 0 {
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-1"}}, tagFamily.Tags[0].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-1"}}, tagFamily.Tags[1].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "endpoint-1"}}, tagFamily.Tags[2].Value)
-
- assert.Equal(t, 2, len(dataPoint.Fields))
- assert.Equal(t, &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 100}}, dataPoint.Fields[0].Value)
- assert.Equal(t, &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 200}}, dataPoint.Fields[1].Value)
- } else {
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "service-2"}}, tagFamily.Tags[0].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "instance-2"}}, tagFamily.Tags[1].Value)
- assert.Equal(t, &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "endpoint-2"}}, tagFamily.Tags[2].Value)
-
- assert.Equal(t, 2, len(dataPoint.Fields))
- assert.Equal(t, &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 200}}, dataPoint.Fields[0].Value)
- assert.Equal(t, &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 500}}, dataPoint.Fields[1].Value)
- }
-
- callCount++
- return "", nil
- }).Times(2)
-
- migrateMeasure(ctx, measure, queryResult, shardNum, replicas, mockSelector, mockClient, l)
-
- assert.Equal(t, 1, queryResult.index)
- assert.Equal(t, 2, callCount, "Expected exactly 2 elements to be processed")
-}
-
-func TestParseGroup(t *testing.T) {
- l := logger.GetLogger("test")
-
- ctrl := gomock.NewController(t)
- defer ctrl.Finish()
-
- tests := []struct {
- group *commonv1.Group
- nodeLabels map[string]string
- name string
- errorMessage string
- nodes []*databasev1.Node
- expectShard uint32
- expectReplicas uint32
- expectError bool
- expectResult bool
- }{
- {
- name: "no resource opts",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: nil,
- },
- nodeLabels: map[string]string{"env": "test"},
- nodes: []*databasev1.Node{},
- expectError: true,
- errorMessage: "no resource opts in group test-group",
- },
- {
- name: "no stages",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: nil,
- },
- },
- nodeLabels: map[string]string{"env": "test"},
- nodes: []*databasev1.Node{},
- expectError: true,
- errorMessage: "no stages in group test-group",
- },
- {
- name: "node matches current stage and there is next stage",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: []*commonv1.LifecycleStage{
- {
- Name: "hot",
- NodeSelector: "env=hot",
- ShardNum: 2,
- },
- {
- Name: "warm",
- NodeSelector: "env=warm",
- ShardNum: 4,
- },
- },
- },
- },
- nodeLabels: map[string]string{"env": "hot"},
- nodes: []*databasev1.Node{
- {
- Metadata: &commonv1.Metadata{
- Name: "node1",
- },
- Labels: map[string]string{"env": "warm"},
- },
- },
- expectResult: true,
- expectShard: 4,
- },
- {
- name: "node matches no stage, use first stage",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: []*commonv1.LifecycleStage{
- {
- Name: "hot",
- NodeSelector: "env=hot",
- ShardNum: 2,
- },
- {
- Name: "warm",
- NodeSelector: "env=warm",
- ShardNum: 4,
- },
- },
- },
- },
- nodeLabels: map[string]string{"env": "unknown"},
- nodes: []*databasev1.Node{
- {
- Metadata: &commonv1.Metadata{
- Name: "node1",
- },
- Labels: map[string]string{"env": "hot"},
- },
- },
- expectResult: true,
- expectShard: 2,
- },
- {
- name: "node matches last stage, no next stage",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: []*commonv1.LifecycleStage{
- {
- Name: "hot",
- NodeSelector: "env=hot",
- ShardNum: 2,
- },
- {
- Name: "warm",
- NodeSelector: "env=warm",
- ShardNum: 4,
- },
- },
- },
- },
- nodeLabels: map[string]string{"env": "warm"},
- nodes: []*databasev1.Node{},
- expectResult: false,
- },
- {
- name: "next stage selector has no matching nodes",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: []*commonv1.LifecycleStage{
- {
- Name: "hot",
- NodeSelector: "env=hot",
- ShardNum: 2,
- },
- {
- Name: "warm",
- NodeSelector: "env=warm",
- ShardNum: 4,
- },
- },
- },
- },
- nodeLabels: map[string]string{"env": "hot"},
- nodes: []*databasev1.Node{
- {
- Metadata: &commonv1.Metadata{
- Name: "node1",
- },
- Labels: map[string]string{"env": "cold"},
- },
- },
- expectError: true,
- errorMessage: "no nodes matched",
- },
- {
- name: "invalid node selector format",
- group: &commonv1.Group{
- Metadata: &commonv1.Metadata{
- Name: "test-group",
- },
- ResourceOpts: &commonv1.ResourceOpts{
- Stages: []*commonv1.LifecycleStage{
- {
- Name: "hot",
- NodeSelector: "invalid-selector",
- ShardNum: 2,
- },
- },
- },
- },
- nodeLabels: map[string]string{"env": "hot"},
- nodes: []*databasev1.Node{},
- expectError: true,
- errorMessage: "no nodes matched",
- },
- }
-
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- mockGroup := schema.NewMockGroup(ctrl)
- mockGroup.EXPECT().ListGroup(gomock.Any()).Return([]*commonv1.Group{}, nil).AnyTimes()
-
- mockRepo := metadata.NewMockRepo(ctrl)
- mockRepo.EXPECT().RegisterHandler("", schema.KindGroup, gomock.Any()).MaxTimes(1)
-
- mockRepo.EXPECT().GroupRegistry().Return(mockGroup).AnyTimes()
- shardNum, replicas, _, selector, client, err := parseGroup(tt.group, tt.nodeLabels, tt.nodes, l, mockRepo)
-
- if tt.expectError {
- require.Error(t, err)
- assert.Contains(t, err.Error(), tt.errorMessage)
- return
- }
-
- if !tt.expectResult {
- require.Nil(t, err)
- require.Nil(t, selector)
- require.Nil(t, client)
- require.Equal(t, uint32(0), shardNum)
- require.Equal(t, uint32(0), replicas)
- return
- }
-
- require.NoError(t, err)
- require.NotNil(t, selector)
- require.NotNil(t, client)
- require.Equal(t, tt.expectShard, shardNum)
- require.Equal(t, tt.expectReplicas, replicas)
- })
- }
-}
diff --git a/banyand/backup/lifecycle/file_migration_visitor.go b/banyand/backup/lifecycle/stream_migration_visitor.go
similarity index 98%
rename from banyand/backup/lifecycle/file_migration_visitor.go
rename to banyand/backup/lifecycle/stream_migration_visitor.go
index c5ea084e..83c8e8b9 100644
--- a/banyand/backup/lifecycle/file_migration_visitor.go
+++ b/banyand/backup/lifecycle/stream_migration_visitor.go
@@ -356,9 +356,7 @@ func (mv *streamMigrationVisitor) VisitElementIndex(segmentTR *timestamp.TimeRan
// calculateTargetShardID maps source shard ID to target shard ID.
func (mv *streamMigrationVisitor) calculateTargetShardID(sourceShardID uint32) uint32 {
- // Simple modulo-based mapping from source shard to target shard
- // This ensures deterministic and balanced distribution
- return sourceShardID % mv.targetShardNum
+ return calculateTargetShardID(sourceShardID, mv.targetShardNum)
}
// streamPartToTargetShard sends part data to all replicas of the target shard.
@@ -486,3 +484,11 @@ func (mv *streamMigrationVisitor) SetStreamElementIndexCount(totalSegmentFiles i
Msg("set stream element index segment count for progress tracking")
}
}
+
+// calculateTargetShardID maps source shard ID to target shard ID.
+// Used by both stream and measure migration visitors.
+func calculateTargetShardID(sourceShardID uint32, targetShardNum uint32) uint32 {
+ // Simple modulo-based mapping from source shard to target shard
+ // This ensures deterministic and balanced distribution
+ return sourceShardID % targetShardNum
+}
diff --git a/banyand/measure/part.go b/banyand/measure/part.go
index 0d885620..cca054f2 100644
--- a/banyand/measure/part.go
+++ b/banyand/measure/part.go
@@ -28,6 +28,7 @@ import (
"github.com/apache/skywalking-banyandb/api/common"
"github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/banyand/queue"
"github.com/apache/skywalking-banyandb/pkg/bytes"
"github.com/apache/skywalking-banyandb/pkg/compress/zstd"
"github.com/apache/skywalking-banyandb/pkg/encoding"
@@ -605,3 +606,77 @@ func partPath(root string, epoch uint64) string {
func partName(epoch uint64) string {
return fmt.Sprintf("%016x", epoch)
}
+
+// CreatePartFileReaderFromPath opens all files in a measure part directory and returns their FileInfo and a cleanup function.
+// Similar to stream.CreatePartFileReaderFromPath but adapted for measure file structure.
+func CreatePartFileReaderFromPath(partPath string, lfs fs.FileSystem) ([]queue.FileInfo, func()) {
+ var files []queue.FileInfo
+ var readers []fs.Reader
+
+ // Core measure files (required files)
+ coreFiles := map[string]string{
+ metaFilename: measureMetaName,
+ primaryFilename: measurePrimaryName,
+ timestampsFilename: measureTimestampsName,
+ fieldValuesFilename: measureFieldValuesName,
+ }
+
+ for filename, streamName := range coreFiles {
+ filePath := path.Join(partPath, filename)
+ reader, err := lfs.OpenFile(filePath)
+ if err != nil {
+ logger.Panicf("cannot open measure file %q: %s", filePath, err)
+ }
+ readers = append(readers, reader)
+ files = append(files, queue.FileInfo{
+ Name: streamName,
+ Reader: reader.SequentialRead(),
+ })
+ }
+
+ // Dynamic tag family files (*.tf and *.tfm)
+ ee := lfs.ReadDir(partPath)
+ for _, e := range ee {
+ if e.IsDir() {
+ continue
+ }
+
+ // Tag family metadata files (.tfm)
+ if filepath.Ext(e.Name()) == tagFamiliesMetadataFilenameExt {
+ tfmPath := path.Join(partPath, e.Name())
+ tfmReader, err := lfs.OpenFile(tfmPath)
+ if err != nil {
+ logger.Panicf("cannot open tag family metadata file %q: %s", tfmPath, err)
+ }
+ readers = append(readers, tfmReader)
+ tagName := removeExt(e.Name(), tagFamiliesMetadataFilenameExt)
+ files = append(files, queue.FileInfo{
+ Name: measureTagMetadataPrefix + tagName,
+ Reader: tfmReader.SequentialRead(),
+ })
+ }
+
+ // Tag family data files (.tf)
+ if filepath.Ext(e.Name()) == tagFamiliesFilenameExt {
+ tfPath := path.Join(partPath, e.Name())
+ tfReader, err := lfs.OpenFile(tfPath)
+ if err != nil {
+ logger.Panicf("cannot open tag family file %q: %s", tfPath, err)
+ }
+ readers = append(readers, tfReader)
+ tagName := removeExt(e.Name(), tagFamiliesFilenameExt)
+ files = append(files, queue.FileInfo{
+ Name: measureTagFamiliesPrefix + tagName,
+ Reader: tfReader.SequentialRead(),
+ })
+ }
+ }
+
+ cleanup := func() {
+ for _, reader := range readers {
+ fs.MustClose(reader)
+ }
+ }
+
+ return files, cleanup
+}
diff --git a/banyand/measure/part_metadata.go b/banyand/measure/part_metadata.go
index 8d961d92..dec52f16 100644
--- a/banyand/measure/part_metadata.go
+++ b/banyand/measure/part_metadata.go
@@ -24,6 +24,7 @@ import (
"github.com/pkg/errors"
"github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/banyand/queue"
"github.com/apache/skywalking-banyandb/pkg/fs"
"github.com/apache/skywalking-banyandb/pkg/logger"
)
@@ -96,3 +97,26 @@ func (pm *partMetadata) mustWriteMetadata(fileSystem fs.FileSystem, partPath str
logger.Panicf("unexpected number of bytes written to %s; got %d; want %d", metadataPath, n, len(metadata))
}
}
+
+// ParsePartMetadata parses the part metadata from the metadata.json file.
+func ParsePartMetadata(fileSystem fs.FileSystem, partPath string) (queue.StreamingPartData, error) {
+ metadataPath := filepath.Join(partPath, metadataFilename)
+ metadata, err := fileSystem.Read(metadataPath)
+ if err != nil {
+ return queue.StreamingPartData{}, errors.WithMessage(err, "cannot read metadata.json")
+ }
+ var pm partMetadata
+ if err := json.Unmarshal(metadata, &pm); err != nil {
+ return queue.StreamingPartData{}, errors.WithMessage(err, "cannot parse metadata.json")
+ }
+
+ return queue.StreamingPartData{
+ ID: pm.ID,
+ CompressedSizeBytes: pm.CompressedSizeBytes,
+ UncompressedSizeBytes: pm.UncompressedSizeBytes,
+ TotalCount: pm.TotalCount,
+ BlocksCount: pm.BlocksCount,
+ MinTimestamp: pm.MinTimestamp,
+ MaxTimestamp: pm.MaxTimestamp,
+ }, nil
+}
diff --git a/banyand/measure/visitor.go b/banyand/measure/visitor.go
new file mode 100644
index 00000000..44edd1ee
--- /dev/null
+++ b/banyand/measure/visitor.go
@@ -0,0 +1,90 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "path/filepath"
+ "strconv"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/pkg/fs"
+ "github.com/apache/skywalking-banyandb/pkg/timestamp"
+)
+
+// Visitor defines the interface for visiting measure components.
+type Visitor interface {
+ // VisitSeries visits the series index directory for a segment.
+ VisitSeries(segmentTR *timestamp.TimeRange, seriesIndexPath string, shardIDs []common.ShardID) error
+ // VisitPart visits a part directory within a shard.
+ VisitPart(segmentTR *timestamp.TimeRange, shardID common.ShardID, partPath string) error
+}
+
+// measureSegmentVisitor adapts Visitor to work with storage.SegmentVisitor.
+type measureSegmentVisitor struct {
+ visitor Visitor
+}
+
+// VisitSeries implements Visitor.
+func (mv *measureSegmentVisitor) VisitSeries(segmentTR *timestamp.TimeRange, seriesIndexPath string, shardIDs []common.ShardID) error {
+ return mv.visitor.VisitSeries(segmentTR, seriesIndexPath, shardIDs)
+}
+
+// VisitShard implements storage.SegmentVisitor.
+func (mv *measureSegmentVisitor) VisitShard(segmentTR *timestamp.TimeRange, shardID common.ShardID, shardPath string) error {
+ // Visit parts within the shard
+ return mv.visitShardParts(segmentTR, shardID, shardPath)
+}
+
+// visitShardParts visits all part directories within a shard.
+func (mv *measureSegmentVisitor) visitShardParts(segmentTR *timestamp.TimeRange, shardID common.ShardID, shardPath string) error {
+ lfs := fs.NewLocalFileSystem()
+ entries := lfs.ReadDir(shardPath)
+
+ for _, entry := range entries {
+ if !entry.IsDir() {
+ continue
+ }
+
+ name := entry.Name()
+ // Check if this is a part directory (16-character hex string)
+ if len(name) != 16 {
+ continue // Skip non-part entries
+ }
+
+ // Validate it's a valid hex string (part ID)
+ if _, err := strconv.ParseUint(name, 16, 64); err != nil {
+ continue // Skip invalid part entries
+ }
+
+ partPath := filepath.Join(shardPath, name)
+ if err := mv.visitor.VisitPart(segmentTR, shardID, partPath); err != nil {
+ return err
+ }
+ }
+
+ return nil
+}
+
+// VisitMeasuresInTimeRange traverses measure segments within the specified time range
+// and calls the visitor methods for parts within shards.
+// This function works directly with the filesystem without requiring a database instance.
+func VisitMeasuresInTimeRange(tsdbRootPath string, timeRange timestamp.TimeRange, visitor Visitor, intervalRule storage.IntervalRule) error {
+ adapter := &measureSegmentVisitor{visitor: visitor}
+ return storage.VisitSegmentsInTimeRange(tsdbRootPath, timeRange, adapter, intervalRule)
+}