You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "suvodeep-pyne (via GitHub)" <gi...@apache.org> on 2025/10/31 18:47:10 UTC
[I] [Feature Request] Enhanced Segment Reload Status Tracking with Failure Details [pinot]
suvodeep-pyne opened a new issue, #17123:
URL: https://github.com/apache/pinot/issues/17123
## Problem Statement
### Current Limitation
The existing segment reload status API (`GET /segments/segmentReloadStatus/{jobId}`) uses timestamp-based heuristics to determine reload success by comparing `segmentLoadTimeMs >= jobSubmissionTime`. This approach has several limitations:
1. **Cannot distinguish between states**: Pending, in-progress, and failed segments all appear as "not successful"
2. **No failure visibility**: When reloads fail, errors are logged but not queryable via API
3. **No error details**: Users must access pod logs to understand why segments failed to reload
4. **False positives**: Segments reloaded for unrelated reasons may be incorrectly counted as successful
### Impact
- **Debugging difficulty**: Users cannot determine which segments failed or why without log access
- **Monitoring gaps**: No programmatic way to alert on reload failures
- **Incomplete observability**: Controllers and clients lack visibility into reload operation health
---
## Proposed Solution
Implement an in-memory status cache on Pinot servers to track per-segment reload status with failure details, enabling comprehensive reload observability through the existing API.
### Design Principles
1. **Memory-efficient**: Bounded cache size with LRU eviction and configurable TTL
2. **Thread-safe**: Handle concurrent reload operations safely
3. **Backward compatible**: Enhance existing APIs without breaking changes
4. **Fail-fast**: Cache is required, not optional - fail at startup if misconfigured
5. **Operationally friendly**: Configurable limits and auto-cleanup
---
## Implementation Plan
### Phase 1: Basic Failure Tracking ✅ (PR #17099)
**Status**: In review
**Scope**:
- In-memory cache tracking failure counts per reload job
- Job ID preservation in Helix messages
- Enhanced server API returning failure counts
- Controller aggregation of failure counts
**Deliverables**:
- `ServerReloadJobStatusCache` with configurable size/TTL
- Server API: `GET /controllerJob/reloadStatus?reloadJobId={id}` returns `failureCount`
- Controller API: `GET /segments/segmentReloadStatus/{jobId}` includes aggregated failures
- Full backward compatibility maintained
**Current PR**: #17099
---
### Phase 2: Per-Segment Status Tracking (Planned)
**Scope**:
- Track individual segment status: `PENDING`, `IN_PROGRESS`, `SUCCESS`, `FAILED`
- Store full error details including:
- Exception class and message
- Complete stack traces
- Log correlation IDs
- Failure timestamps
- Enhanced response with per-segment breakdown
**API Enhancement**:
```json
{
"successCount": 7,
"failedCount": 2,
"inProgressCount": 1,
"segmentStatuses": [
{
"segmentName": "seg1",
"status": "SUCCESS",
"startTimeMs": 1234567890,
"endTimeMs": 1234567900
},
{
"segmentName": "seg2",
"status": "FAILED",
"errorSummary": "IOException: Connection timeout",
"stackTrace": "java.io.IOException: ...",
"logCorrelationId": "2024-01-15T10:23:45.123Z"
}
]
}
```
**Memory Budget**:
- Typical case: ~230 MB for 10,000 jobs with 100 segments each (1% failure rate)
- Worst case: ~1.1 GB for 10% failure rate
- Impact: 0.7-6.9% of typical 16-32 GB server heap (acceptable)
---
### Phase 3: Complete Failure Path Coverage (Planned)
**Scope**:
- Fix gaps in failure tracking coverage:
- Single segment reload path (currently untracked)
- Config fetch failures (occurs before instrumented try-catch)
- Semaphore acquire failures (partial coverage)
- Add safety net at message handler level to catch all failure types
**Coverage Matrix** (Current State):
| Failure Type | Single Segment | Batch Reload | Coverage |
|--------------|----------------|--------------|----------|
| Config fetch | ❌ Not tracked | ❌ Not tracked | 0% |
| Semaphore acquire | ❌ Not tracked | ✅ Tracked | Partial |
| Download failures | ❌ Not tracked | ✅ Tracked | Partial |
| Index building | ❌ Not tracked | ✅ Tracked | Partial |
---
## Design Details
### Cache Structure
```java
Cache<String, ConcurrentHashMap<String, ReloadSegmentStatus>>
// Key: jobId (UUID)
// Value: Map of segmentName -> status
class ReloadSegmentStatus {
ReloadStatus status; // PENDING, IN_PROGRESS, SUCCESS, FAILED
long startTimeMs;
long endTimeMs;
String errorSummary; // Exception class + message (200 chars max)
String stackTrace; // Full stack trace (null if success)
String logCorrelationId; // ISO timestamp for log lookup
}
```
### Configuration
```properties
# Server configuration
pinot.server.reload.status.cache.max.size=10000
pinot.server.reload.status.cache.ttl.days=30
pinot.server.reload.status.error.summary.max.chars=200
```
### Thread Safety
- **Guava Cache** protects outer map (jobId → segment map)
- **ConcurrentHashMap** protects inner map (segment → status)
- State transition validation prevents invalid updates
---
## Benefits
1. **Improved Debugging**: Users can query failure details via API instead of searching logs
2. **Better Monitoring**: Programmatic access to failure counts and error types
3. **Operational Visibility**: Clear insight into reload job health across the cluster
4. **Reduced MTTR**: Faster incident diagnosis with accessible error details
5. **API Consistency**: Job-based tracking aligns with job ID already returned by reload APIs
---
## Alternatives Considered
1. **ZooKeeper-based persistence**: Rejected due to write overhead and ZK load concerns
2. **Compressed stack traces**: Rejected for simplicity; full traces fit acceptable memory budget
3. **Optional cache**: Rejected; making cache required simplifies code and improves reliability
---
## Open Questions
1. Should failed entries have longer TTL than successful ones for debugging?
2. Should we expose cache admin APIs (clear, stats) for operational management?
3. What metrics/alerts should be added for reload failure monitoring?
---
## References
- Related PR: #17099 (Phase 1 implementation)
- Design Principle: Memory-first with bounded growth
- Inspiration: Similar tracking in other distributed systems (Kubernetes Job status, Spark task tracking)
---
## Community Feedback Welcome
We welcome feedback on:
- Phase priorities and scope
- API response format
- Memory budget concerns
- Alternative approaches
- Additional use cases
---
**Note**: This is a multi-phase enhancement designed to incrementally improve reload observability while maintaining backward compatibility and operational stability. Phase 1 provides immediate value with minimal risk, while subsequent phases build toward comprehensive failure tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Feature Request] Enhanced Segment Reload Status Tracking with Failure Details [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17123: [Feature Request] Enhanced Segment Reload Status Tracking with Failure Details
URL: https://github.com/apache/pinot/issues/17123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Feature Request] Enhanced Segment Reload Status Tracking with Failure Details [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17123:
URL: https://github.com/apache/pinot/issues/17123#issuecomment-4676144952
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Feature Request] Enhanced Segment Reload Status Tracking with Failure Details [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17123:
URL: https://github.com/apache/pinot/issues/17123#issuecomment-4317211589
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org