You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by kh...@apache.org on 2025/01/10 18:49:20 UTC
(pinot) branch pauseless_ingestion_without_failure_scenarios updated (a97847f586 -> 81ea9eeb6f)
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a change to branch pauseless_ingestion_without_failure_scenarios
in repository https://gitbox.apache.org/repos/asf/pinot.git
from a97847f586 Changing the way to enable pausless post introduction of multi-stream ingestion by moving it out of streamConfigMap
add 4f92b6da2e 1. Removing pauseless from indexingConfig 2. Fixing few formatting issues. 3. Fixing few comments for better readability. 4. Removing unused code: contructors, constants etc
add 9fae1374d4 Fixing pausless integration tests
add b04cad4885 Removing checks for CRC to that of status checks
add 2495730436 Refactoring FSM to reduce code duplication
add f4b1420a7f Improving the download segment for realtime tables by: 1. Relying on segment status to make the call on whether to use waited download instead of tableConfig. - This is needed for pauseless tables as the pauseless can be disabled by changing the table config - Segment status is a better decision criteria. 2. Checking presence of an ONLINE server before attempting peer download to prevent waiting for exponential backoff.
new 81ea9eeb6f Fix status check for downloads to also include UPLOADED status
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../common/metadata/segment/SegmentZKMetadata.java | 4 +-
.../common/utils/PauselessConsumptionUtils.java | 11 --
.../realtime/BlockingSegmentCompletionFSM.java | 44 +++--
.../realtime/PauselessSegmentCompletionFSM.java | 186 +++++----------------
.../core/data/manager/BaseTableDataManager.java | 52 +++---
.../realtime/PauselessSegmentCommitter.java | 5 -
.../realtime/RealtimeSegmentDataManager.java | 5 +-
.../manager/realtime/RealtimeTableDataManager.java | 62 +++++--
.../pinot/core/util/PeerServerSegmentFinder.java | 2 +-
.../PauselessRealtimeIngestionIntegrationTest.java | 10 +-
.../pinot/spi/config/table/IndexingConfig.java | 9 -
11 files changed, 162 insertions(+), 228 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
(pinot) 01/01: Fix status check for downloads to also include UPLOADED status
Posted by kh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch pauseless_ingestion_without_failure_scenarios
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit 81ea9eeb6f26ae91919664bb00eff78b760ae6a2
Author: Kartik Khare <kh...@Kartiks-MacBook-Pro.local>
AuthorDate: Sat Jan 11 00:18:30 2025 +0530
Fix status check for downloads to also include UPLOADED status
---
.../apache/pinot/core/data/manager/BaseTableDataManager.java | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index cafd1aa50b..c1462ec5b9 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -653,7 +653,7 @@ public abstract class BaseTableDataManager implements TableDataManager {
- Continue loading the segment from the index directory.
*/
boolean shouldDownload =
- forceDownload || (zkMetadata.getStatus() == CommonConstants.Segment.Realtime.Status.DONE && !hasSameCRC(
+ forceDownload || (isSegmentStatusCompleted(zkMetadata) && !hasSameCRC(
zkMetadata, localMetadata));
if (shouldDownload) {
// Create backup directory to handle failure of segment reloading.
@@ -717,6 +717,11 @@ public abstract class BaseTableDataManager implements TableDataManager {
_logger.info("Reloaded segment: {}", segmentName);
}
+ private boolean isSegmentStatusCompleted(SegmentZKMetadata zkMetadata) {
+ return zkMetadata.getStatus() == CommonConstants.Segment.Realtime.Status.DONE
+ || zkMetadata.getStatus() == CommonConstants.Segment.Realtime.Status.UPLOADED;
+ }
+
private boolean canReuseExistingDirectoryForReload(SegmentZKMetadata segmentZKMetadata, String currentSegmentTier,
SegmentDirectory segmentDirectory, IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
@@ -1011,8 +1016,7 @@ public abstract class BaseTableDataManager implements TableDataManager {
Then:
We need to fall back to downloading the segment from deep storage to load it.
*/
- if (segmentMetadata == null || (zkMetadata.getStatus() == CommonConstants.Segment.Realtime.Status.DONE
- && !hasSameCRC(zkMetadata, segmentMetadata))) {
+ if (segmentMetadata == null || (isSegmentStatusCompleted(zkMetadata) && !hasSameCRC(zkMetadata, segmentMetadata))) {
if (segmentMetadata == null) {
_logger.info("Segment: {} does not exist", segmentName);
} else if (!hasSameCRC(zkMetadata, segmentMetadata)) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org