You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "KKcorps (via GitHub)" <gi...@apache.org> on 2025/01/27 04:21:19 UTC
[PR] Pauseless reingestion #3: Disaster Recovery with Reingestion [pinot]
KKcorps opened a new pull request, #14920:
URL: https://github.com/apache/pinot/pull/14920
This PR adds support for Disaster recovery for Pauseless Ingestion along with Reingestion.
During Pauseless ingestion, a ONLINE segment can wind up in an ERROR state if its commit fails due to server restart and there are no other replicas. Currently in pinot, there is no way to recover from such failures.
This PR adds a solution where we use `RealtimeSegmentValidationManager` to check segments in this state and then trigger a reingestion for such segments.
The reingestion is done in the servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14920:
URL: https://github.com/apache/pinot/pull/14920#issuecomment-2621270356
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14920?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `24.24242%` with `25 lines` in your changes missing coverage. Please review.
> Project coverage is 56.08%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`e84788a`)](https://app.codecov.io/gh/apache/pinot/commit/e84788a519a104fd2f942584ccbb06b250d3242d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1643 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/14920?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2FFileUploadDownloadClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | 0.00% | [20 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeTableDataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fserver%2Frealtime%2FServerSegmentCompletionProtocolHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2FBaseTableDataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | 88.88% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (e84788a). Click for more details.
>
> <details><summary>HEAD has 48 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (e84788a) |
>|------|------|------|
>|integration|7|0|
>|integration2|3|0|
>|temurin|12|2|
>|java-21|7|2|
>|skip-bytebuffers-true|3|1|
>|skip-bytebuffers-false|7|1|
>|unittests|5|2|
>|java-11|5|0|
>|unittests2|3|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #14920 +/- ##
============================================
- Coverage 61.75% 56.08% -5.67%
- Complexity 207 711 +504
============================================
Files 2436 2119 -317
Lines 133233 114468 -18765
Branches 20636 18406 -2230
============================================
- Hits 82274 64203 -18071
- Misses 44911 45033 +122
+ Partials 6048 5232 -816
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.08% <24.24%> (-5.54%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.01% <24.24%> (-5.74%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.06% <24.24%> (+28.33%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.08% <24.24%> (-5.67%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.08% <24.24%> (-5.66%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.08% <24.24%> (+9.19%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/14920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/14920?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940993753
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
+ try {
+ _consumer.start(_startOffset);
+ _currentOffset = _startOffset;
+ TransformPipeline.Result reusedResult = new TransformPipeline.Result();
+ while (!_shouldStop.get() && _currentOffset.compareTo(_endOffset) < 0) {
+ // Fetch messages
+ MessageBatch messageBatch = _consumer.fetchMessages(_currentOffset, _fetchTimeoutMs);
+
+ int messageCount = messageBatch.getMessageCount();
+
+ for (int i = 0; i < messageCount; i++) {
+ if (_shouldStop.get()) {
+ break;
+ }
+ StreamMessage streamMessage = messageBatch.getStreamMessage(i);
+ if (streamMessage.getMetadata() != null && streamMessage.getMetadata().getOffset() != null
+ && streamMessage.getMetadata().getOffset().compareTo(_endOffset) >= 0) {
+ _shouldStop.set(true);
+ _logger.info("Reached end offset: {} for partition group: {}", _endOffset, _partitionGroupId);
+ break;
+ }
+
+ // Decode message
+ StreamDataDecoderResult decodedResult = _decoder.decode(streamMessage);
+ if (decodedResult.getException() == null) {
+ // Index message
+ GenericRow row = decodedResult.getResult();
+
+ _transformPipeline.processRow(row, reusedResult);
+
+ List<GenericRow> transformedRows = reusedResult.getTransformedRows();
+
+ // TODO: Do enrichment and transforms before indexing
+ for (GenericRow transformedRow : transformedRows) {
+ _realtimeSegment.index(transformedRow, streamMessage.getMetadata());
+ _numRowsIndexed++;
+ }
+ } else {
+ _logger.warn("Failed to decode message at offset {}: {}", _currentOffset, decodedResult.getException());
+ }
+ }
+
+ _currentOffset = messageBatch.getOffsetOfNextBatch();
+ }
+ _isSuccess = true;
+ } catch (Exception e) {
+ _logger.error("Exception in consumer thread", e);
+ _consumptionException = e;
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ _consumer.close();
+ } catch (Exception e) {
+ _logger.warn("Failed to close consumer", e);
+ }
+ _isDoneConsuming.set(true);
+ }
+ }
+ }
+
+ public void stopConsumption() {
+ _shouldStop.set(true);
+ if (_consumerThread.isAlive()) {
+ _consumerThread.interrupt();
+ try {
+ _consumerThread.join();
+ } catch (InterruptedException e) {
+ _logger.warn("Interrupted while waiting for consumer thread to finish");
+ }
+ }
+ }
+
+ @Override
+ public MutableSegment getSegment() {
+ return _realtimeSegment;
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ protected void doDestroy() {
+ _realtimeSegment.destroy();
+ FileUtils.deleteQuietly(_resourceTmpDir);
+ FileUtils.deleteQuietly(_resourceDataDir);
+ }
+
+ @Override
+ public void doOffload() {
+ stopConsumption();
+ }
+
+ public boolean isDoneConsuming() {
+ return _isDoneConsuming.get();
+ }
+
+ public boolean isSuccess() {
+ return _isSuccess;
+ }
+
+ public Throwable getConsumptionException() {
+ return _consumptionException;
+ }
+
+ @VisibleForTesting
+ public SegmentBuildDescriptor buildSegmentInternal() throws Exception {
+ _logger.info("Building segment from {} to {}", _startOffset, _currentOffset);
+ final long lockAcquireTimeMillis = now();
+ // Build a segment from in-memory rows.
+ // Use a temporary directory
+ Path tempSegmentFolder = null;
+ try {
+ tempSegmentFolder =
+ java.nio.file.Files.createTempDirectory(_resourceTmpDir.toPath(), "tmp-" + _segmentNameStr + "-");
+ } catch (IOException e) {
+ _logger.error("Failed to create temporary directory for segment build", e);
+ return null;
+ }
+
+ SegmentZKPropsConfig segmentZKPropsConfig = new SegmentZKPropsConfig();
+ segmentZKPropsConfig.setStartOffset(_startOffset.toString());
+ segmentZKPropsConfig.setEndOffset(_endOffset.toString());
+
+ // Build the segment
+ RealtimeSegmentConverter converter =
+ new RealtimeSegmentConverter(_realtimeSegment, segmentZKPropsConfig, tempSegmentFolder.toString(),
+ _schema, _tableNameWithType, _tableConfig, _segmentZKMetadata.getSegmentName(),
+ _tableConfig.getIndexingConfig().isNullHandlingEnabled());
+ try {
+ converter.build(null, _serverMetrics);
+ } catch (Exception e) {
+ _logger.error("Failed to build segment", e);
+ FileUtils.deleteQuietly(tempSegmentFolder.toFile());
+ return null;
+ }
+ final long buildTimeMillis = now() - lockAcquireTimeMillis;
Review Comment:
We are not acquiring any lock. Why is the variable named like that ?
We are by-passing the number of parallel build constraints that we have for each server.
I understand that this will be breached by maximum number of parallel reingestions i.e. we can have
`50 + Num of re-ingestions allowed in parallel ` number of segments that can be build in parallel.
For now `Num of re-ingestions allowed in parallel` is restricted to 8 is not a concern but once this is made configurable it can put pressure on server resources.
@KKcorps Is there a reason to not attain that lock ?
cc: @swaminathanmanish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1946106811
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentWriter(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ Semaphore segBuildSemaphore, ServerMetrics serverMetrics)
+ throws Exception {
+ _segBuildSemaphore = segBuildSemaphore;
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
Review Comment:
synced offline and using data dir now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1943016498
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING state
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
Review Comment:
Moved it to pinot llc class itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1972260409
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,464 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.base.Preconditions;
+import java.io.Closeable;
+import java.io.File;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Segment writer to ingest streaming data from a start offset to an end offset.
+ *
+ * TODO:
+ * 1. Clean up this class and only keep the necessary parts.
+ * 2. Use a different segment impl for better performance because it doesn't need to serve queries.
+ */
+public class StatelessRealtimeSegmentWriter implements Closeable {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final String SEGMENT_STATS_FILE_NAME = "segment-stats.ser";
+ public static final String RESOURCE_TMP_DIR_PREFIX = "resourceTmpDir_";
+ public static final String RESOURCE_DATA_DIR_PREFIX = "resourceDataDir_";
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+
+ public StatelessRealtimeSegmentWriter(SegmentZKMetadata segmentZKMetadata, IndexLoadingConfig indexLoadingConfig,
+ @Nullable Semaphore segBuildSemaphore)
+ throws Exception {
+ Preconditions.checkNotNull(indexLoadingConfig.getTableConfig(), "Table config must be set in index loading config");
+ Preconditions.checkNotNull(indexLoadingConfig.getSchema(), "Schema must be set in index loading config");
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentZKMetadata.getSegmentName());
+
+ _segmentName = segmentZKMetadata.getSegmentName();
+ _partitionGroupId = llcSegmentName.getPartitionGroupId();
Review Comment:
This will be incompatible with multi-stream ingestion. We need to use the streamPartitionGroupId to do the ingestion. Refer to the RealtimeSegmentDataManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1936058894
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
+
+ // Send immediate success response with jobId
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ Response response = Response.ok(immediateResponse).build();
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ SimpleRealtimeSegmentDataManager manager = new SimpleRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
Review Comment:
I was talking about at just the instant just after flag is set false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939094922
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
+ }
+ }
+ }
+
+ /**
+ * Picks one 'alive' server among a set of servers that are supposed to host the segment,
+ * e.g. by checking if Helix says it is enabled or if it appears in the live instance list.
+ * This is a simple example; adapt to your environment’s definition of “alive.”
+ */
+ private String findAliveServerToReIngest(Set<String> candidateServers) {
+ // Get the current live instances from Helix
+ HelixDataAccessor helixDataAccessor = _helixManager.getHelixDataAccessor();
+ PropertyKey.Builder keyBuilder = helixDataAccessor.keyBuilder();
+ List<String> liveInstances = helixDataAccessor.getChildNames(keyBuilder.liveInstances());
+ try {
+ // This should ideally handle https scheme as well
+ BiMap<String, String> instanceToEndpointMap =
+ _helixResourceManager.getDataInstanceAdminEndpoints(candidateServers);
+
+ if (instanceToEndpointMap.isEmpty()) {
+ LOGGER.warn("No instance data admin endpoints found for servers: {}", candidateServers);
+ return null;
+ }
+
+ for (String server : candidateServers) {
+ if (liveInstances.contains(server)) {
+ return instanceToEndpointMap.get(server);
Review Comment:
Even when a null is returned there is a possibility that another server satisfied both the conditions but whose end-point couldn't be returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938922613
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
Review Comment:
Want to understand the scenario when a segment is marked ONLINE in the IS for one of the servers while not for others.
What happens to the IS when a server is disabled ? Is it removed from the IS or marked OFFLINE ?
What would happen to the IS in this case ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939694120
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
+ }
+ }
+ }
+
+ /**
+ * Picks one 'alive' server among a set of servers that are supposed to host the segment,
+ * e.g. by checking if Helix says it is enabled or if it appears in the live instance list.
+ * This is a simple example; adapt to your environment’s definition of “alive.”
+ */
+ private String findAliveServerToReIngest(Set<String> candidateServers) {
+ // Get the current live instances from Helix
+ HelixDataAccessor helixDataAccessor = _helixManager.getHelixDataAccessor();
+ PropertyKey.Builder keyBuilder = helixDataAccessor.keyBuilder();
+ List<String> liveInstances = helixDataAccessor.getChildNames(keyBuilder.liveInstances());
+ try {
+ // This should ideally handle https scheme as well
+ BiMap<String, String> instanceToEndpointMap =
+ _helixResourceManager.getDataInstanceAdminEndpoints(candidateServers);
+
+ if (instanceToEndpointMap.isEmpty()) {
+ LOGGER.warn("No instance data admin endpoints found for servers: {}", candidateServers);
+ return null;
+ }
+
+ for (String server : candidateServers) {
+ if (liveInstances.contains(server)) {
+ return instanceToEndpointMap.get(server);
Review Comment:
Good catch ideally though instanceToEndpointMap should be superset of liveInstances
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941388026
##########
pinot-core/src/main/java/org/apache/pinot/server/realtime/ServerSegmentCompletionProtocolHandler.java:
##########
@@ -100,6 +100,14 @@ public AuthProvider getAuthProvider() {
return _authProvider;
}
+ public String getProtocol() {
+ return _protocol;
+ }
+
+ public Integer getControllerHttpsPort() {
+ return _controllerHttpsPort;
+ }
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941507341
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
Yeah, I am also not happy with this. The thing I am trying to solve for is basically making segment refresh succeed for reingesion
Without this check it fails on the following line:
```
Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
"Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
```
Segment refresh is triggered whenever a segment is uploaded which is what reingestion is doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1942150405
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
We probably need a new API to handle re-ingested segment. It is not regular segment push. With a new API we can also set the status to `DONE`, and then trigger the reset from the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940875087
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
Review Comment:
This is just a note: Flag `isReadyToConsumeData` has not been added here as this flag caters to partial upsert and dedup table that rely on ingestion order.
Pauseless does not support either of the table types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939093329
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
Review Comment:
This should be triggered even in the case when either one of the replicas is in ERROR.
Also, consider the following scenario.
1. Server 1 fails to download the segment.
2. Server 2 somehow uploads the segment and completed the COMMIT_END_METADATA call. In this case, the segment status will be `DONE`.
Also, why do we need this status check ?
If a segment is in ERROR state in any of the segments and the download url is present, I think it's safe to reset the segment.
We can create another function that takes care of such segments that are in ERROR state and have a `non-empty `download url
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939198758
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
Review Comment:
I think this approach of rate limiting a re-ingestion request is risky and there are no immediate retries on the controller.
Concern: The segment will be missing till the next job runs.
I propose the following:
Assumption: The number of re-ingestions are relatively less i.e. max one segment per partition
Proposal: The server can maintain a queue to read from and start ingestion. We can still limit the number of parallel ingestion by taking the semaphore approach thus reducing CPU and memory requirements.
The server, before starting the ingestion, can check whether the download url is present or not. If not present, it can continue with the reingestion otherwise return an error.
Additionally, we can add a flag for force `re-ingestion` to ensure that we server re-ingests irrespective of the presence/ absence of url.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939752971
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
Review Comment:
This already happens in segment level validation
https://github.com/apache/pinot/pull/14217/files
```
if (_segmentAutoResetOnErrorAtValidation) {
_pinotHelixResourceManager.resetSegments(realtimeTableName, null, true);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1943015601
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
Review Comment:
Changed this, now reset happens as part of new API. No need for this code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940790579
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
Review Comment:
I think it's better to update the tests than put a condition here. Not sure whether we will run into this.
Seems fine for now though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941389686
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
+ }, CHECK_INTERVAL_MS, UPLOAD_END_TIMEOUT_MS, 0);
+
+ // Trigger segment reset
+ HttpClient httpClient = HttpClient.getInstance();
+ Map<String, String> headersMap = headers.stream()
+ .collect(Collectors.toMap(Header::getName, Header::getValue));
+ resetSegment(httpClient, controllerUrl, tableNameWithType, segmentName, null, headersMap);
+
+ LOGGER.info("Re-ingested segment {} uploaded successfully", segmentName);
+ } finally {
+ manager.offload();
+ manager.destroy();
+ }
+ }
+
+ private void waitForCondition(
+ Function<Void, Boolean> condition, long checkIntervalMs, long timeoutMs, long gracePeriodMs) {
+ long endTime = System.currentTimeMillis() + timeoutMs;
+
+ // Adding grace period before starting the condition checks
+ if (gracePeriodMs > 0) {
Review Comment:
Will leave it here, useful for future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939691471
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
Review Comment:
fixed comment, wasn't considering placeholder url was being considered in old code but had changed it to only COMMITTING state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938837675
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
@KKcorps I only have a concern with the way we trigger this i.e. waiting for the segment to go in ERROR state.
Do you think we should add an API to check this ? If a Mutable/ Immutable segment manager exists on any of the servers then it means that the segment might still being build. In case it's not, then we can safely trigger the re-ingestion.
@Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939658249
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
Review Comment:
That can happen even in non pauseless scenario right? It should already be handled. Ideally we need to simply reset the segment in this scenario without reingestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1936067264
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
> reingestion runs extremely infrequently (once per few hours)
Isn't this a concern as well? Users might not be ok with data missing for a long time
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
> reingestion runs extremely infrequently (once per few hours)
Isn't this a concern as well? Users might not be ok with data missing for a long time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940998922
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
+ try {
+ _consumer.start(_startOffset);
+ _currentOffset = _startOffset;
+ TransformPipeline.Result reusedResult = new TransformPipeline.Result();
+ while (!_shouldStop.get() && _currentOffset.compareTo(_endOffset) < 0) {
+ // Fetch messages
+ MessageBatch messageBatch = _consumer.fetchMessages(_currentOffset, _fetchTimeoutMs);
+
+ int messageCount = messageBatch.getMessageCount();
+
+ for (int i = 0; i < messageCount; i++) {
+ if (_shouldStop.get()) {
+ break;
+ }
+ StreamMessage streamMessage = messageBatch.getStreamMessage(i);
+ if (streamMessage.getMetadata() != null && streamMessage.getMetadata().getOffset() != null
+ && streamMessage.getMetadata().getOffset().compareTo(_endOffset) >= 0) {
+ _shouldStop.set(true);
+ _logger.info("Reached end offset: {} for partition group: {}", _endOffset, _partitionGroupId);
+ break;
+ }
+
+ // Decode message
+ StreamDataDecoderResult decodedResult = _decoder.decode(streamMessage);
+ if (decodedResult.getException() == null) {
+ // Index message
+ GenericRow row = decodedResult.getResult();
+
+ _transformPipeline.processRow(row, reusedResult);
+
+ List<GenericRow> transformedRows = reusedResult.getTransformedRows();
+
+ // TODO: Do enrichment and transforms before indexing
+ for (GenericRow transformedRow : transformedRows) {
+ _realtimeSegment.index(transformedRow, streamMessage.getMetadata());
+ _numRowsIndexed++;
+ }
+ } else {
+ _logger.warn("Failed to decode message at offset {}: {}", _currentOffset, decodedResult.getException());
+ }
+ }
+
+ _currentOffset = messageBatch.getOffsetOfNextBatch();
+ }
+ _isSuccess = true;
+ } catch (Exception e) {
+ _logger.error("Exception in consumer thread", e);
+ _consumptionException = e;
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ _consumer.close();
+ } catch (Exception e) {
+ _logger.warn("Failed to close consumer", e);
+ }
+ _isDoneConsuming.set(true);
+ }
+ }
+ }
+
+ public void stopConsumption() {
+ _shouldStop.set(true);
+ if (_consumerThread.isAlive()) {
+ _consumerThread.interrupt();
+ try {
+ _consumerThread.join();
+ } catch (InterruptedException e) {
+ _logger.warn("Interrupted while waiting for consumer thread to finish");
+ }
+ }
+ }
+
+ @Override
+ public MutableSegment getSegment() {
+ return _realtimeSegment;
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ protected void doDestroy() {
+ _realtimeSegment.destroy();
+ FileUtils.deleteQuietly(_resourceTmpDir);
+ FileUtils.deleteQuietly(_resourceDataDir);
+ }
+
+ @Override
+ public void doOffload() {
+ stopConsumption();
+ }
+
+ public boolean isDoneConsuming() {
+ return _isDoneConsuming.get();
+ }
+
+ public boolean isSuccess() {
+ return _isSuccess;
+ }
+
+ public Throwable getConsumptionException() {
+ return _consumptionException;
+ }
+
+ @VisibleForTesting
+ public SegmentBuildDescriptor buildSegmentInternal() throws Exception {
+ _logger.info("Building segment from {} to {}", _startOffset, _currentOffset);
+ final long lockAcquireTimeMillis = now();
+ // Build a segment from in-memory rows.
+ // Use a temporary directory
+ Path tempSegmentFolder = null;
+ try {
+ tempSegmentFolder =
+ java.nio.file.Files.createTempDirectory(_resourceTmpDir.toPath(), "tmp-" + _segmentNameStr + "-");
+ } catch (IOException e) {
+ _logger.error("Failed to create temporary directory for segment build", e);
+ return null;
+ }
+
+ SegmentZKPropsConfig segmentZKPropsConfig = new SegmentZKPropsConfig();
+ segmentZKPropsConfig.setStartOffset(_startOffset.toString());
+ segmentZKPropsConfig.setEndOffset(_endOffset.toString());
+
+ // Build the segment
+ RealtimeSegmentConverter converter =
+ new RealtimeSegmentConverter(_realtimeSegment, segmentZKPropsConfig, tempSegmentFolder.toString(),
+ _schema, _tableNameWithType, _tableConfig, _segmentZKMetadata.getSegmentName(),
+ _tableConfig.getIndexingConfig().isNullHandlingEnabled());
+ try {
+ converter.build(null, _serverMetrics);
+ } catch (Exception e) {
+ _logger.error("Failed to build segment", e);
+ FileUtils.deleteQuietly(tempSegmentFolder.toFile());
+ return null;
+ }
+ final long buildTimeMillis = now() - lockAcquireTimeMillis;
+
+ File dataDir = _resourceDataDir;
Review Comment:
Nit: There is no need for this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941464061
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
+ try {
+ _consumer.start(_startOffset);
+ _currentOffset = _startOffset;
+ TransformPipeline.Result reusedResult = new TransformPipeline.Result();
+ while (!_shouldStop.get() && _currentOffset.compareTo(_endOffset) < 0) {
+ // Fetch messages
+ MessageBatch messageBatch = _consumer.fetchMessages(_currentOffset, _fetchTimeoutMs);
+
+ int messageCount = messageBatch.getMessageCount();
+
+ for (int i = 0; i < messageCount; i++) {
+ if (_shouldStop.get()) {
+ break;
+ }
+ StreamMessage streamMessage = messageBatch.getStreamMessage(i);
+ if (streamMessage.getMetadata() != null && streamMessage.getMetadata().getOffset() != null
+ && streamMessage.getMetadata().getOffset().compareTo(_endOffset) >= 0) {
+ _shouldStop.set(true);
+ _logger.info("Reached end offset: {} for partition group: {}", _endOffset, _partitionGroupId);
+ break;
+ }
+
+ // Decode message
+ StreamDataDecoderResult decodedResult = _decoder.decode(streamMessage);
+ if (decodedResult.getException() == null) {
+ // Index message
+ GenericRow row = decodedResult.getResult();
+
+ _transformPipeline.processRow(row, reusedResult);
+
+ List<GenericRow> transformedRows = reusedResult.getTransformedRows();
+
+ // TODO: Do enrichment and transforms before indexing
+ for (GenericRow transformedRow : transformedRows) {
+ _realtimeSegment.index(transformedRow, streamMessage.getMetadata());
+ _numRowsIndexed++;
+ }
+ } else {
+ _logger.warn("Failed to decode message at offset {}: {}", _currentOffset, decodedResult.getException());
+ }
+ }
+
+ _currentOffset = messageBatch.getOffsetOfNextBatch();
+ }
+ _isSuccess = true;
+ } catch (Exception e) {
+ _logger.error("Exception in consumer thread", e);
+ _consumptionException = e;
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ _consumer.close();
+ } catch (Exception e) {
+ _logger.warn("Failed to close consumer", e);
+ }
+ _isDoneConsuming.set(true);
+ }
+ }
+ }
+
+ public void stopConsumption() {
+ _shouldStop.set(true);
+ if (_consumerThread.isAlive()) {
+ _consumerThread.interrupt();
+ try {
+ _consumerThread.join();
+ } catch (InterruptedException e) {
+ _logger.warn("Interrupted while waiting for consumer thread to finish");
+ }
+ }
+ }
+
+ @Override
+ public MutableSegment getSegment() {
+ return _realtimeSegment;
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ protected void doDestroy() {
+ _realtimeSegment.destroy();
+ FileUtils.deleteQuietly(_resourceTmpDir);
+ FileUtils.deleteQuietly(_resourceDataDir);
+ }
+
+ @Override
+ public void doOffload() {
+ stopConsumption();
+ }
+
+ public boolean isDoneConsuming() {
+ return _isDoneConsuming.get();
+ }
+
+ public boolean isSuccess() {
+ return _isSuccess;
+ }
+
+ public Throwable getConsumptionException() {
+ return _consumptionException;
+ }
+
+ @VisibleForTesting
+ public SegmentBuildDescriptor buildSegmentInternal() throws Exception {
+ _logger.info("Building segment from {} to {}", _startOffset, _currentOffset);
+ final long lockAcquireTimeMillis = now();
+ // Build a segment from in-memory rows.
+ // Use a temporary directory
+ Path tempSegmentFolder = null;
+ try {
+ tempSegmentFolder =
+ java.nio.file.Files.createTempDirectory(_resourceTmpDir.toPath(), "tmp-" + _segmentNameStr + "-");
+ } catch (IOException e) {
+ _logger.error("Failed to create temporary directory for segment build", e);
+ return null;
+ }
+
+ SegmentZKPropsConfig segmentZKPropsConfig = new SegmentZKPropsConfig();
+ segmentZKPropsConfig.setStartOffset(_startOffset.toString());
+ segmentZKPropsConfig.setEndOffset(_endOffset.toString());
+
+ // Build the segment
+ RealtimeSegmentConverter converter =
+ new RealtimeSegmentConverter(_realtimeSegment, segmentZKPropsConfig, tempSegmentFolder.toString(),
+ _schema, _tableNameWithType, _tableConfig, _segmentZKMetadata.getSegmentName(),
+ _tableConfig.getIndexingConfig().isNullHandlingEnabled());
+ try {
+ converter.build(null, _serverMetrics);
+ } catch (Exception e) {
+ _logger.error("Failed to build segment", e);
+ FileUtils.deleteQuietly(tempSegmentFolder.toFile());
+ return null;
+ }
+ final long buildTimeMillis = now() - lockAcquireTimeMillis;
Review Comment:
Good catch, I was not attaining it mostly so that we can reingest faster.
But you are right, we should not overload the servers.
Adding it back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1944074719
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentWriter(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ Semaphore segBuildSemaphore, ServerMetrics serverMetrics)
+ throws Exception {
+ _segBuildSemaphore = segBuildSemaphore;
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
Review Comment:
In which scenarios it won't work? The reason I am not using data dir is to avoid scenarios that mess up actual data due to overwrites
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/ZKOperator.java:
##########
@@ -72,6 +72,17 @@ public void completeSegmentOperations(String tableNameWithType, SegmentMetadata
@Nullable String sourceDownloadURIStr, String segmentDownloadURIStr, @Nullable String crypterName,
long segmentSizeInBytes, boolean enableParallelPushProtection, boolean allowRefresh, HttpHeaders headers)
throws Exception {
+ completeSegmentOperations(tableNameWithType, segmentMetadata, uploadType, finalSegmentLocationURI, segmentFile,
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940830134
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
+ }, CHECK_INTERVAL_MS, UPLOAD_END_TIMEOUT_MS, 0);
+
+ // Trigger segment reset
+ HttpClient httpClient = HttpClient.getInstance();
+ Map<String, String> headersMap = headers.stream()
+ .collect(Collectors.toMap(Header::getName, Header::getValue));
+ resetSegment(httpClient, controllerUrl, tableNameWithType, segmentName, null, headersMap);
+
+ LOGGER.info("Re-ingested segment {} uploaded successfully", segmentName);
+ } finally {
+ manager.offload();
+ manager.destroy();
+ }
+ }
+
+ private void waitForCondition(
+ Function<Void, Boolean> condition, long checkIntervalMs, long timeoutMs, long gracePeriodMs) {
+ long endTime = System.currentTimeMillis() + timeoutMs;
+
+ // Adding grace period before starting the condition checks
+ if (gracePeriodMs > 0) {
Review Comment:
Nit: The gracePeriod is never used i.e. both the function calls pass 0 as the grace period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940974264
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
+ try {
+ _consumer.start(_startOffset);
+ _currentOffset = _startOffset;
+ TransformPipeline.Result reusedResult = new TransformPipeline.Result();
+ while (!_shouldStop.get() && _currentOffset.compareTo(_endOffset) < 0) {
+ // Fetch messages
+ MessageBatch messageBatch = _consumer.fetchMessages(_currentOffset, _fetchTimeoutMs);
+
+ int messageCount = messageBatch.getMessageCount();
+
+ for (int i = 0; i < messageCount; i++) {
+ if (_shouldStop.get()) {
+ break;
+ }
+ StreamMessage streamMessage = messageBatch.getStreamMessage(i);
+ if (streamMessage.getMetadata() != null && streamMessage.getMetadata().getOffset() != null
+ && streamMessage.getMetadata().getOffset().compareTo(_endOffset) >= 0) {
+ _shouldStop.set(true);
+ _logger.info("Reached end offset: {} for partition group: {}", _endOffset, _partitionGroupId);
+ break;
+ }
+
+ // Decode message
+ StreamDataDecoderResult decodedResult = _decoder.decode(streamMessage);
+ if (decodedResult.getException() == null) {
+ // Index message
+ GenericRow row = decodedResult.getResult();
+
+ _transformPipeline.processRow(row, reusedResult);
+
+ List<GenericRow> transformedRows = reusedResult.getTransformedRows();
+
+ // TODO: Do enrichment and transforms before indexing
+ for (GenericRow transformedRow : transformedRows) {
+ _realtimeSegment.index(transformedRow, streamMessage.getMetadata());
+ _numRowsIndexed++;
Review Comment:
This is not used anywhere ?
Is there a reason to include this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941501560
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING state
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
Review Comment:
the reason I have kept it here so that I can utilise correct http client with all the ssl settings. What would be appropriate class to move this to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1942248666
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
Review Comment:
Are we not logging this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1942241681
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
+ serverHostPort = serverHostPort.replace(HTTP + "://", "");
+ }
+
+ String serverHost = serverHostPort.split(":")[0];
+ String serverPort = serverHostPort.split(":")[1];
+
+ URI reIngestUri = getURI(scheme, serverHost, Integer.parseInt(serverPort), REINGEST_SEGMENT_PATH);
+
+ Map<String, Object> requestJson = new HashMap<>();
+ requestJson.put("tableNameWithType", tableNameWithType);
+ requestJson.put("segmentName", segmentName);
+
+ String jsonPayload = JsonUtils.objectToString(requestJson);
+ SimpleHttpResponse response =
+ HttpClient.wrapAndThrowHttpException(_httpClient.sendJsonPostRequest(reIngestUri, jsonPayload));
+
+ // Check that we got a 2xx response
+ int statusCode = response.getStatusCode();
+ if (statusCode / 100 != 2) {
+ throw new IOException(String.format("Failed POST to %s, HTTP %d: %s",
Review Comment:
Was the code updated here ?
I see that the comment has been resolved but no changes in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on PR #14920:
URL: https://github.com/apache/pinot/pull/14920#issuecomment-2683545261
This is a great feature. And I believe we could expand its usage beyond failure recovery.
E.g. we could combine the feature with this proposal https://github.com/apache/pinot/issues/14815 for better ingestion freshness guarantee.
- Have a similar **ingestAdhoc** endpoint to, instead of _re-ingesting_, **ingest a segment based on the input partition and offset**.
- Hooked together with the proposal in https://github.com/apache/pinot/issues/14815. The controller skips the offset based on freshness status, and also spawns the asynchronous ingestAdhoc job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless reingestion #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1930746482
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2195,6 +2198,139 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in DONE status with a missing download URL, but also
Review Comment:
Shouldn't this be:
`Re-ingests segments that are in `ONLINE` status with a missing download URL, but also`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1936058894
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
+
+ // Send immediate success response with jobId
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ Response response = Response.ok(immediateResponse).build();
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ SimpleRealtimeSegmentDataManager manager = new SimpleRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
Review Comment:
I was talking about at just the instant just after flag is set false. (Just looking from as a server API perspective ignoring however frequently this API is called)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1944074284
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.realtime.writer.StatelessRealtimeSegmentWriter;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment/{segmentName}")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(@PathParam("segmentName") String segmentName) {
+ // if segment is not in LLC format, return error
+ if (!LLCSegmentName.isLLCSegment(segmentName)) {
+ throw new WebApplicationException("Segment name is not in LLC format: " + segmentName,
+ Response.Status.BAD_REQUEST);
+ }
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ String tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(llcSegmentName.getTableName());
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ RealtimeTableDataManager tableDataManager =
+ (RealtimeTableDataManager) instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
Review Comment:
Not possible. Ideally though in multi-stream only broker and topic names should differ across stream configs, rest everything should remain the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940355486
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -442,6 +446,10 @@ public IdealState getIdealState(String realtimeTableName) {
}
}
+ public ExternalView getExternalView(String realtimeTableName) {
+ return _helixResourceManager.getTableExternalView(realtimeTableName);
+ }
+
Review Comment:
(minor) Let's inline this and remove this method. The behavior is not consistent with `getIdealState()` which can cause confusion
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
Review Comment:
(minor) When the table type is known, let's directly use the type for readability
```suggestion
public void reIngestSegmentsWithErrorState(String realtimeTableName) {
```
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1243,7 +1243,7 @@ protected boolean buildSegmentAndReplace()
return true;
}
- private void closeStreamConsumers() {
+ protected void closeStreamConsumers() {
Review Comment:
(minor) Seems not needed, or do you plan to add a test?
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,39 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
Review Comment:
Suggest just making it `POST /reIngestSegment/{segmentName}` which is easier to use.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,39 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
Review Comment:
(minor) table name is implicit from the LLC segment name, and we may simplify this API
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
Review Comment:
Can you modify while looping over the list? I believe this will throw exception
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
Review Comment:
(minor) I'd check this before checking ERROR segment because this one is cheaper
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
Review Comment:
When there are `ONLINE` replica, ideally we should reset the `ERROR` replica. Do we rely on validation manager for that?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING state
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
Review Comment:
This doesn't belong to upload/download category. Can you add this method to a different class?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
Review Comment:
`externalView` could be `null`. We need null check then return
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
Review Comment:
Can we move this class to `segment-local` so that it can be reused by minion?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING state
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
Review Comment:
We shouldn't need this check. If a server is `ERROR` in EV, it should be live.
We also want to randomly pick the server to avoid creating hotspot
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
Review Comment:
You may merge this loop and the next one, and simply `continue` here
##########
pinot-core/src/main/java/org/apache/pinot/server/realtime/ServerSegmentCompletionProtocolHandler.java:
##########
@@ -100,6 +100,14 @@ public AuthProvider getAuthProvider() {
return _authProvider;
}
+ public String getProtocol() {
+ return _protocol;
+ }
+
+ public Integer getControllerHttpsPort() {
+ return _controllerHttpsPort;
+ }
+
Review Comment:
The logic of handling upload re-ingested segment should be within this class. Can you move the logic?
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -638,6 +637,16 @@ private long getDownloadTimeOutMilliseconds(@Nullable TableConfig tableConfig) {
}).orElse(DEFAULT_SEGMENT_DOWNLOAD_TIMEOUT_MS);
}
+ protected RealtimeSegmentDataManager createRealtimeSegmentDataManager(SegmentZKMetadata zkMetadata,
Review Comment:
(minor) Annotate `@VisibleForTesting`
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
Review Comment:
I don't think we need a data manager here. We are not using it to serve query, so I'd suggest simplifying this class and only keep the ingestion related part
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
When will crc match? We shouldn't replace a segment multiple times (e.g. somehow 2 servers trying to re-ingest)
##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/FailureInjectingRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests.realtime;
+
+import java.util.concurrent.Semaphore;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.retry.AttemptsExceededException;
+import org.apache.pinot.spi.utils.retry.RetriableOperationException;
+
+
+/**
+ * A specialized RealtimeSegmentDataManager that lets us inject a forced failure
+ * in the commit step, which occurs strictly after the segmentConsumed message.
+ */
+public class FailureInjectingRealtimeSegmentDataManager extends RealtimeSegmentDataManager {
Review Comment:
(minor) Keep all failure injecting class in the same package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939232541
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
Review Comment:
Maybe using
```
private static final ExecutorService REINGESTION_EXECUTOR =
new ThreadPoolExecutor(
MAX_PARALLEL_EXECUTIONS, // core pool size
MAX_PARALLEL_EXECUTIONS, // maximum pool size (same as core to ensure fixed size)
0L, TimeUnit.MILLISECONDS, // keep-alive time
new LinkedBlockingQueue<>(), // unbounded queue for holding submitted tasks
new ThreadFactoryBuilder()
.setNameFormat("reingestion-worker-%d")
.build()
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940867017
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
Review Comment:
Let's log like the way we do in normal ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940863270
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
Review Comment:
I am concerned about the impact on the server metrics. This object is used by normal ingestion as well.
From an overview, it doesn't seem to be so risky as the metrics emitted would be for the segment, table or server.
The issue with metrics happens when we have 2 consumers for the same partition.
@KKcorps i recall some PR for subbu that mentioned the issue and why we don't allow 2 consumers for same partition.
For re-ingestion we would be having multiple consumers for the same partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1935509704
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/SimpleRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class SimpleRealtimeSegmentDataManager extends SegmentDataManager {
+
Review Comment:
Unfortunately that is not possible as these methods diverge a lot from main ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1935539149
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
+
+ // Send immediate success response with jobId
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ Response response = Response.ok(immediateResponse).build();
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ SimpleRealtimeSegmentDataManager manager = new SimpleRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
Review Comment:
No, because it will get returned when trying to set this to true right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940959863
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
Review Comment:
cc: @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939085910
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
+ }
+ }
+ }
+
+ /**
+ * Picks one 'alive' server among a set of servers that are supposed to host the segment,
+ * e.g. by checking if Helix says it is enabled or if it appears in the live instance list.
+ * This is a simple example; adapt to your environment’s definition of “alive.”
+ */
+ private String findAliveServerToReIngest(Set<String> candidateServers) {
+ // Get the current live instances from Helix
+ HelixDataAccessor helixDataAccessor = _helixManager.getHelixDataAccessor();
+ PropertyKey.Builder keyBuilder = helixDataAccessor.keyBuilder();
+ List<String> liveInstances = helixDataAccessor.getChildNames(keyBuilder.liveInstances());
+ try {
+ // This should ideally handle https scheme as well
+ BiMap<String, String> instanceToEndpointMap =
+ _helixResourceManager.getDataInstanceAdminEndpoints(candidateServers);
+
+ if (instanceToEndpointMap.isEmpty()) {
+ LOGGER.warn("No instance data admin endpoints found for servers: {}", candidateServers);
+ return null;
+ }
+
+ for (String server : candidateServers) {
+ if (liveInstances.contains(server)) {
Review Comment:
```suggestion
if (liveInstances.contains(server) && instanceToEndpointMap.containsKey(server)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939679641
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
Review Comment:
Yeah I found it to not have anything in some local tests. Hence kept it like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939664013
##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -176,6 +177,12 @@ private void runSegmentLevelValidation(TableConfig tableConfig) {
// Update the total document count gauge
_validationMetrics.updateTotalDocumentCountGauge(realtimeTableName, computeTotalDocumentCount(segmentsZKMetadata));
+ boolean isPauselessConsumptionEnabled = PauselessConsumptionUtils.isPauselessEnabled(tableConfig);
+
+ if (isPauselessConsumptionEnabled) {
Review Comment:
That's a good point, let me moved it outside segment level validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938837011
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
This is expected to be a rare scenario where the servers restart at the time of building the segment leading to loss of in-memory segment data and requiring this job.
I think for RealtimeSegmentValidationManager the frequency can be kept low to ensure the gap does not last for long.
Also, this can be manually triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939110550
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
I am not able to come up with a better approach to achieve this but this does not seem the right way to cater to re-ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940830134
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
+ }, CHECK_INTERVAL_MS, UPLOAD_END_TIMEOUT_MS, 0);
+
+ // Trigger segment reset
+ HttpClient httpClient = HttpClient.getInstance();
+ Map<String, String> headersMap = headers.stream()
+ .collect(Collectors.toMap(Header::getName, Header::getValue));
+ resetSegment(httpClient, controllerUrl, tableNameWithType, segmentName, null, headersMap);
+
+ LOGGER.info("Re-ingested segment {} uploaded successfully", segmentName);
+ } finally {
+ manager.offload();
+ manager.destroy();
+ }
+ }
+
+ private void waitForCondition(
+ Function<Void, Boolean> condition, long checkIntervalMs, long timeoutMs, long gracePeriodMs) {
+ long endTime = System.currentTimeMillis() + timeoutMs;
+
+ // Adding grace period before starting the condition checks
+ if (gracePeriodMs > 0) {
Review Comment:
Nit: The gracePeriod is never used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1946082135
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
Review Comment:
RESET would work for pauseless provided we have peer download enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1943018091
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
Cool, added a new API called `/segment/completeReingestion` that handles this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938839462
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
Also, I feel that we lack any way of triggering reingestion without the segment going in ERROR state. We have to wait for ERROR state even in case of manual trigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938938285
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
Review Comment:
Why is this check needed given that `segmentsInErrorState` comes from `segmentToInstanceCurrentStateMap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938930531
##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -176,6 +177,12 @@ private void runSegmentLevelValidation(TableConfig tableConfig) {
// Update the total document count gauge
_validationMetrics.updateTotalDocumentCountGauge(realtimeTableName, computeTotalDocumentCount(segmentsZKMetadata));
+ boolean isPauselessConsumptionEnabled = PauselessConsumptionUtils.isPauselessEnabled(tableConfig);
+
+ if (isPauselessConsumptionEnabled) {
Review Comment:
As I see, this function is called within `runSegmentLevelValidation`. Under normal circumstances, this call shouldn't fetch all the segment ZK metadata as not many segments should be in `ERROR`.
Moreover, segment level validations run at a reduced frequency as they fetch segment ZK metadata for all the segments e.g. missing download url.
These segment level validations don't impact the query or ingestion. Other things done by RealtimeSegmentValidationManager are ensuring that the ingestion works fine even in case of commit protocol failure.
I feel re-ingestion is `critical` as this will impact queries (`partial results` in case of `single stage` query engine and `complete failure` in case of `multi-stage` query engine) and hence should not be under segment level validations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938946714
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
Review Comment:
In what scenario would we have a placeholder url.
We don't have a placeholder url in case of `COMMIT_END_METADATA` call failure ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939672741
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
Review Comment:
Hmm, I don't want to follow this cause ideally I would like controller to schedule reingestion on another server if one is already occupied, rather than filling up its queue. This logic is not present in controller currently but can be added.
Ideal way to solve this is how we do it for minions where we maintain a queue in ZK and available servers can pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939696837
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
Review Comment:
Yeah this is not needed since I am performing reset call from server itself now.
For the scenario you mentioned, why do we not RESET the segments currently though in such situation without pauseless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941000154
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
Review Comment:
Synced offline and we decided it's better to go with queue based model than outright rejecting requests to have faster reingestion during scenarios such as server restarts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941044749
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
Review Comment:
`When there are ONLINE replica, ideally we should reset the ERROR replica. Do we rely on validation manager for that?` yes that is already a part of validation manager https://github.com/apache/pinot/pull/14217/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941381089
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
Review Comment:
We need refresh to finish before triggering reset, otherwise the reset operation simply fails saying a refresh is in progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941436572
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1944609599
##########
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##########
@@ -180,7 +180,10 @@ public enum ControllerGauge implements AbstractMetrics.Gauge {
// segment when the partition is first detected).
COMMITTING_SEGMENT_SIZE("committingSegmentSize", false),
- TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false);
+ TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false),
+
+ // Number of in progress segment reingestion
+ SEGMENT_REINGESTION_IN_PROGRESS("segmentReingestionInProgress", true);
Review Comment:
Renamed it to `SEGMENT_REINGESTION_UPLOAD_IN_PROGRESS`
And it should work, we do the same to track segment upload as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1946074533
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
Review Comment:
We do reset the segments: https://github.com/apache/pinot/pull/14217/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #14920:
URL: https://github.com/apache/pinot/pull/14920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless reingestion #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on PR #14920:
URL: https://github.com/apache/pinot/pull/14920#issuecomment-2616072758
So this check is only performed when RealtimeSegmentValidationManager job runs right? If yes than should this be logic be part of different job with higher frequency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938845451
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
Review Comment:
Nit: Is there a possibility that the `serverHostPort` doesn't contain the protocol ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938875488
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
Review Comment:
Do we need to wait for the segment to be in `ERROR` state for all the servers.
What would happen in this case:
1. Server 1 goes in ERROR state.
2. Server 2 is able to build it eventually.
Server 1 stays in ERROR state as it's not reset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939663253
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
Review Comment:
No when a table is disabled, it simply adds a flag `enabled: false` in ideal state. everything else remains the same. the segments are then marked offline (not deleted)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1935222572
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
Review Comment:
(minor) maybe let's breakup all these similar validations into separate method (will be useful for unit testing as well).
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
+
+ // Send immediate success response with jobId
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ Response response = Response.ok(immediateResponse).build();
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ SimpleRealtimeSegmentDataManager manager = new SimpleRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
Review Comment:
Won't there be race condition here when same segments is again scheduled to be re-ingested?
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
Review Comment:
(minor) should add to RUNNING_JOBS after submitting to executor.
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/SimpleRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class SimpleRealtimeSegmentDataManager extends SegmentDataManager {
+
Review Comment:
Won't this create code debt in future maintaining both SimpleRealtimeSegmentDataManager and RealtimeSegmentDataManager?
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
These states are in-memory hence curious about the case when controllers sends re-ingest request to another server for same segment.
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/SimpleRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class SimpleRealtimeSegmentDataManager extends SegmentDataManager {
+
Review Comment:
Maybe we should leverage RealtimeSegmentDataManager methods in this class
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
+ .entity("Reingestion jobs parallel limit " + MAX_PARALLEL_REINGESTIONS + " reached.").build();
+ }
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+ RUNNING_JOBS.put(jobId, job);
+
+ // Send immediate success response with jobId
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ Response response = Response.ok(immediateResponse).build();
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ SimpleRealtimeSegmentDataManager manager = new SimpleRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ // Return immediately with the jobId
+ return response;
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(SimpleRealtimeSegmentDataManager manager, String segmentName, String tableNameWithType,
+ IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), 1000, CONSUMPTION_END_TIMEOUT_MS, 0);
Review Comment:
intervals and timeouts seems quite small
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1935512278
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,526 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.SimpleRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = 300000L;
+ public static final long UPLOAD_END_TIMEOUT_MS = 300000L;
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
Review Comment:
That's a good point. Currently it will get executed. However that scenario is unlikely as reingestion runs extremely infrequently (once per few hours)
One possible solution could be ZK
Another would be to keep track on controller side instead of server side.
Will check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1943877931
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/ZKOperator.java:
##########
@@ -170,6 +190,26 @@ public void completeSegmentsOperations(String tableNameWithType, FileUploadType
processExistingSegments(tableNameWithType, uploadType, enableParallelPushProtection, headers, existingSegmentsList);
}
+ public void updateReingestedSegmentZKMetadata(String tableNameWithType, SegmentMetadata segmentMetadata,
Review Comment:
Not used?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/ZKOperator.java:
##########
@@ -72,6 +72,17 @@ public void completeSegmentOperations(String tableNameWithType, SegmentMetadata
@Nullable String sourceDownloadURIStr, String segmentDownloadURIStr, @Nullable String crypterName,
long segmentSizeInBytes, boolean enableParallelPushProtection, boolean allowRefresh, HttpHeaders headers)
throws Exception {
+ completeSegmentOperations(tableNameWithType, segmentMetadata, uploadType, finalSegmentLocationURI, segmentFile,
Review Comment:
Let's not coupling the logic of reingestion into current regular segment complete handling. Currently the logic is very hard to read, and we are coupling reset with reingestion, which is not correct.
We can add a new method `completeReingestedSegmentOperations()`. Reingested segment holds different assumptions then regular uploaded segment.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentWriter(String segmentName, String tableNameWithType, int partitionGroupId,
Review Comment:
A lot of parameters are not needed and can be derived. Passing `SegmentZKMetadata`, `IndexLoadingConfig`, `@Nullable Semaphore` should be enough. We don't want to pass `ServerMetrics` because this can potentially run in minion
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
Review Comment:
Let's clean this up. A lot of fields are not required
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2103,146 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment/[segmentName]
Review Comment:
```suggestion
* POST http://[serverURL]/reingestSegment/[segmentName]
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2103,146 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment/[segmentName]
+ * Request body (JSON):
+ *
+ * @param realtimeTableName The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String realtimeTableName) {
Review Comment:
Keep the name consistent, same for other places
```suggestion
public void reingestSegmentsWithErrorState(String realtimeTableName) {
```
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.realtime.writer.StatelessRealtimeSegmentWriter;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
Review Comment:
If the reingestion failed, this server will reject future reingestion request. We need to remove the entry when the reingestion finishes
##########
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##########
@@ -180,7 +180,10 @@ public enum ControllerGauge implements AbstractMetrics.Gauge {
// segment when the partition is first detected).
COMMITTING_SEGMENT_SIZE("committingSegmentSize", false),
- TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false);
+ TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false),
+
+ // Number of in progress segment reingestion
+ SEGMENT_REINGESTION_IN_PROGRESS("segmentReingestionInProgress", true);
Review Comment:
This is a little bit confusing. It is not re-ingestion in progress, but re-ingested segment upload in progress. Also it is attached to a async handler. Does it give correct tracking in async handler?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java:
##########
@@ -965,6 +972,41 @@ public Response revertReplaceSegments(
}
}
+ @POST
+ @ManagedAsync
+ @Produces(MediaType.APPLICATION_JSON)
+ @Consumes(MediaType.MULTIPART_FORM_DATA)
+ @Path("segment/completeReingestion")
Review Comment:
Follow the existing format for path:
`/segments/reingested"`
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentWriter(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ Semaphore segBuildSemaphore, ServerMetrics serverMetrics)
+ throws Exception {
+ _segBuildSemaphore = segBuildSemaphore;
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
Review Comment:
I think we should pass in a data dir as the working dir. It might not always work to use temp dir
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.realtime.writer.StatelessRealtimeSegmentWriter;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
Review Comment:
Let's use `max(nCPUs / 4, 1)` as the default
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.realtime.writer.StatelessRealtimeSegmentWriter;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
Review Comment:
We usually use `Executors.newFixedThreadPool()` to bound the thread usage. Any benefit of manually constructing the `ThreadPoolExecutor`?
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.realtime.writer.StatelessRealtimeSegmentWriter;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment/{segmentName}")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(@PathParam("segmentName") String segmentName) {
+ // if segment is not in LLC format, return error
+ if (!LLCSegmentName.isLLCSegment(segmentName)) {
+ throw new WebApplicationException("Segment name is not in LLC format: " + segmentName,
+ Response.Status.BAD_REQUEST);
+ }
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ String tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(llcSegmentName.getTableName());
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ RealtimeTableDataManager tableDataManager =
+ (RealtimeTableDataManager) instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
Review Comment:
Is there a way to know which stream config to use to generate the segment?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,131 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
Review Comment:
Can we pull that logic out and handle it differently for pauseless table? For pauseless table, reset doesn't really work. Also that shouldn't be segment level validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1944074844
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,582 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentWriter {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentWriter(String segmentName, String tableNameWithType, int partitionGroupId,
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1946087815
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
It's `/segments/reingested` right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940866516
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
Review Comment:
```suggestion
_consumerThread = new Thread(new PartitionConsumer(), _segmentName);
_logger.info("Created new consumer thread {} for {}", _consumerThread, this);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940951429
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
Review Comment:
How can this impact the snapshot.
The segment has been marked `ONLINE`.
@klsince do you see any concern here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1942252443
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
Review Comment:
Have we tested the implications of passing null ? I think this might create issues since this object in passed around in build etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1943015002
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
Review Comment:
This should not be affected as the next consuming segment should already take care of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1972268037
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -0,0 +1,464 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.realtime.writer;
+
+import com.google.common.base.Preconditions;
+import java.io.Closeable;
+import java.io.File;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Segment writer to ingest streaming data from a start offset to an end offset.
+ *
+ * TODO:
+ * 1. Clean up this class and only keep the necessary parts.
+ * 2. Use a different segment impl for better performance because it doesn't need to serve queries.
+ */
+public class StatelessRealtimeSegmentWriter implements Closeable {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final String SEGMENT_STATS_FILE_NAME = "segment-stats.ser";
+ public static final String RESOURCE_TMP_DIR_PREFIX = "resourceTmpDir_";
+ public static final String RESOURCE_DATA_DIR_PREFIX = "resourceDataDir_";
+
+ private final Semaphore _segBuildSemaphore;
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+
+ public StatelessRealtimeSegmentWriter(SegmentZKMetadata segmentZKMetadata, IndexLoadingConfig indexLoadingConfig,
+ @Nullable Semaphore segBuildSemaphore)
+ throws Exception {
+ Preconditions.checkNotNull(indexLoadingConfig.getTableConfig(), "Table config must be set in index loading config");
+ Preconditions.checkNotNull(indexLoadingConfig.getSchema(), "Schema must be set in index loading config");
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentZKMetadata.getSegmentName());
+
+ _segmentName = segmentZKMetadata.getSegmentName();
+ _partitionGroupId = llcSegmentName.getPartitionGroupId();
+ _segBuildSemaphore = segBuildSemaphore;
+ _tableNameWithType = TableNameBuilder.forType(TableType.REALTIME).tableNameWithType(llcSegmentName.getTableName());
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = indexLoadingConfig.getTableConfig();
+ _schema = indexLoadingConfig.getSchema();
+ String tableDataDir = indexLoadingConfig.getTableDataDir() == null ? FileUtils.getTempDirectory().getAbsolutePath()
+ : indexLoadingConfig.getTableDataDir();
+ File reingestionDir = new File(tableDataDir, "reingestion");
+ _resourceTmpDir =
+ new File(reingestionDir, RESOURCE_TMP_DIR_PREFIX + _segmentName + "_" + System.currentTimeMillis());
+ _resourceDataDir =
+ new File(reingestionDir, RESOURCE_DATA_DIR_PREFIX + _segmentName + "_" + System.currentTimeMillis());
+ FileUtils.deleteQuietly(_resourceTmpDir);
+ FileUtils.deleteQuietly(_resourceDataDir);
+ FileUtils.forceMkdir(_resourceTmpDir);
+ FileUtils.forceMkdir(_resourceDataDir);
+
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentWriter.class.getName() + "_" + _segmentName);
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(_tableConfig).get(0);
Review Comment:
Same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless reingestion #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1930920440
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2195,6 +2198,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
Review Comment:
Reset segment does not work when the SegmentDataManager is missing on the server. Consider the following scenario:
A segment has missing url. The server hosting these segments restart and the segment goes in ERROR state in EV.
The re-ingestion updates the ZKMetadata and the reset segment message is sent.
The server does not have any SegmentDataManager instance for the segment and hence the reset does not work.
```
protected void doReplaceSegment(String segmentName)
throws Exception {
SegmentDataManager segmentDataManager = _segmentDataManagerMap.get(segmentName);
if (segmentDataManager != null) {
SegmentZKMetadata zkMetadata = fetchZKMetadata(segmentName);
IndexLoadingConfig indexLoadingConfig = fetchIndexLoadingConfig();
indexLoadingConfig.setSegmentTier(zkMetadata.getTier());
replaceSegmentIfCrcMismatch(segmentDataManager, zkMetadata, indexLoadingConfig);
} else {
_logger.warn("Failed to find segment: {}, skipping replacing it", segmentName);
}
}
```
A ran the above code and found the following error.
[upsertMeetupRsvp_with_dr_2_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_40] Failed to find segment: upsertMeetupRsvp_with_dr_2__0__57__20250127T0745Z, skipping replacing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1940948236
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/StatelessRealtimeSegmentDataManager.java:
##########
@@ -0,0 +1,575 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources.reingestion.utils;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.TransformPipeline;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.spi.MutableSegment;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
+import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
+import org.apache.pinot.spi.config.table.SegmentZKPropsConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.MessageBatch;
+import org.apache.pinot.spi.stream.PartitionGroupConsumer;
+import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDataDecoder;
+import org.apache.pinot.spi.stream.StreamDataDecoderImpl;
+import org.apache.pinot.spi.stream.StreamDataDecoderResult;
+import org.apache.pinot.spi.stream.StreamMessage;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.retry.RetryPolicies;
+import org.apache.pinot.spi.utils.retry.RetryPolicy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Simplified Segment Data Manager for ingesting data from a start offset to an end offset.
+ */
+public class StatelessRealtimeSegmentDataManager extends SegmentDataManager {
+
+ private static final int DEFAULT_CAPACITY = 100_000;
+ private static final int DEFAULT_FETCH_TIMEOUT_MS = 5000;
+ public static final FileUploadDownloadClient FILE_UPLOAD_DOWNLOAD_CLIENT = new FileUploadDownloadClient();
+
+ private final String _segmentName;
+ private final String _tableNameWithType;
+ private final int _partitionGroupId;
+ private final String _segmentNameStr;
+ private final SegmentZKMetadata _segmentZKMetadata;
+ private final TableConfig _tableConfig;
+ private final Schema _schema;
+ private final StreamConfig _streamConfig;
+ private final StreamPartitionMsgOffsetFactory _offsetFactory;
+ private final StreamConsumerFactory _consumerFactory;
+ private StreamMetadataProvider _partitionMetadataProvider;
+ private final PartitionGroupConsumer _consumer;
+ private final StreamDataDecoder _decoder;
+ private final MutableSegmentImpl _realtimeSegment;
+ private final File _resourceTmpDir;
+ private final File _resourceDataDir;
+ private final Logger _logger;
+ private Thread _consumerThread;
+ private final AtomicBoolean _shouldStop = new AtomicBoolean(false);
+ private final AtomicBoolean _isDoneConsuming = new AtomicBoolean(false);
+ private final StreamPartitionMsgOffset _startOffset;
+ private final StreamPartitionMsgOffset _endOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
+ private volatile int _numRowsIndexed = 0;
+ private final String _segmentStoreUriStr;
+ private final int _fetchTimeoutMs;
+ private final TransformPipeline _transformPipeline;
+ private volatile boolean _isSuccess = false;
+ private volatile Throwable _consumptionException;
+ private final ServerMetrics _serverMetrics;
+
+ public StatelessRealtimeSegmentDataManager(String segmentName, String tableNameWithType, int partitionGroupId,
+ SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig, Schema schema,
+ IndexLoadingConfig indexLoadingConfig, StreamConfig streamConfig, String startOffsetStr, String endOffsetStr,
+ ServerMetrics serverMetrics)
+ throws Exception {
+
+ _segmentName = segmentName;
+ _tableNameWithType = tableNameWithType;
+ _partitionGroupId = partitionGroupId;
+ _segmentZKMetadata = segmentZKMetadata;
+ _tableConfig = tableConfig;
+ _schema = schema;
+ _segmentStoreUriStr = indexLoadingConfig.getSegmentStoreURI();
+ _streamConfig = streamConfig;
+ _resourceTmpDir = new File(FileUtils.getTempDirectory(), "resourceTmpDir_" + System.currentTimeMillis());
+ _resourceDataDir = new File(FileUtils.getTempDirectory(), "resourceDataDir_" + System.currentTimeMillis());;
+ _serverMetrics = serverMetrics;
+ _logger = LoggerFactory.getLogger(StatelessRealtimeSegmentDataManager.class.getName() + "_" + _segmentName);
+
+ _offsetFactory = StreamConsumerFactoryProvider.create(_streamConfig).createStreamMsgOffsetFactory();
+ _startOffset = _offsetFactory.create(startOffsetStr);
+ _endOffset = _offsetFactory.create(endOffsetStr);
+
+ String clientId = getClientId();
+
+ // Temp dirs
+ _resourceTmpDir.mkdirs();
+ _resourceDataDir.mkdirs();
+
+ _consumerFactory = StreamConsumerFactoryProvider.create(_streamConfig);
+ _partitionMetadataProvider = _consumerFactory.createPartitionMetadataProvider(clientId, _partitionGroupId);
+ _segmentNameStr = _segmentZKMetadata.getSegmentName();
+
+ // Create a simple PartitionGroupConsumptionStatus
+ PartitionGroupConsumptionStatus partitionGroupConsumptionStatus =
+ new PartitionGroupConsumptionStatus(_partitionGroupId, 0, _startOffset, null, null);
+
+ _consumer = _consumerFactory.createPartitionGroupConsumer(clientId, partitionGroupConsumptionStatus);
+
+ // Initialize decoder
+ Set<String> fieldsToRead = IngestionUtils.getFieldsForRecordExtractor(_tableConfig, _schema);
+ _decoder = createDecoder(fieldsToRead);
+
+ // Fetch capacity from indexLoadingConfig or use default
+ int capacity = streamConfig.getFlushThresholdRows();
+ if (capacity <= 0) {
+ capacity = DEFAULT_CAPACITY;
+ }
+
+ // Fetch average number of multi-values from indexLoadingConfig
+ int avgNumMultiValues = indexLoadingConfig.getRealtimeAvgMultiValueCount();
+
+ // Load stats history, here we are using the same stats while as the RealtimeSegmentDataManager so that we are
+ // much more efficient in allocating buffers. It also works with empty file
+ String tableDataDir = indexLoadingConfig.getInstanceDataManagerConfig() != null
+ ? indexLoadingConfig.getInstanceDataManagerConfig().getInstanceDataDir() + File.separator + _tableNameWithType
+ : _resourceTmpDir.getAbsolutePath();
+ File statsHistoryFile = new File(tableDataDir, "segment-stats.ser");
+ RealtimeSegmentStatsHistory statsHistory = RealtimeSegmentStatsHistory.deserialzeFrom(statsHistoryFile);
+
+ // Initialize mutable segment with configurations
+ RealtimeSegmentConfig.Builder realtimeSegmentConfigBuilder =
+ new RealtimeSegmentConfig.Builder(indexLoadingConfig).setTableNameWithType(_tableNameWithType)
+ .setSegmentName(_segmentName).setStreamName(_streamConfig.getTopicName())
+ .setSegmentZKMetadata(_segmentZKMetadata).setStatsHistory(statsHistory).setSchema(_schema)
+ .setCapacity(capacity).setAvgNumMultiValues(avgNumMultiValues)
+ .setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation())
+ .setFieldConfigList(tableConfig.getFieldConfigList()).setConsumerDir(_resourceDataDir.getAbsolutePath())
+ .setMemoryManager(
+ new MmapMemoryManager(FileUtils.getTempDirectory().getAbsolutePath(), _segmentNameStr, _serverMetrics));
+
+ setPartitionParameters(realtimeSegmentConfigBuilder, _tableConfig.getIndexingConfig().getSegmentPartitionConfig());
+
+ _realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfigBuilder.build(), _serverMetrics);
+
+ _transformPipeline = new TransformPipeline(tableConfig, schema);
+
+ // Initialize fetch timeout
+ _fetchTimeoutMs =
+ _streamConfig.getFetchTimeoutMillis() > 0 ? _streamConfig.getFetchTimeoutMillis() : DEFAULT_FETCH_TIMEOUT_MS;
+ }
+
+ private String getClientId() {
+ return _tableNameWithType + "-" + _partitionGroupId;
+ }
+
+ public void startConsumption() {
+ // Start the consumer thread
+ _consumerThread = new Thread(new PartitionConsumer(), _segmentName);
+ _consumerThread.start();
+ }
+
+ private StreamDataDecoder createDecoder(Set<String> fieldsToRead)
+ throws Exception {
+ AtomicReference<StreamDataDecoder> localStreamDataDecoder = new AtomicReference<>();
+ RetryPolicy retryPolicy = RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 1.2f);
+ retryPolicy.attempt(() -> {
+ try {
+ StreamMessageDecoder streamMessageDecoder = createMessageDecoder(fieldsToRead);
+ localStreamDataDecoder.set(new StreamDataDecoderImpl(streamMessageDecoder));
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Failed to create StreamMessageDecoder. Retrying...", e);
+ return false;
+ }
+ });
+ return localStreamDataDecoder.get();
+ }
+
+ /**
+ * Creates a {@link StreamMessageDecoder} using properties in {@link StreamConfig}.
+ *
+ * @param fieldsToRead The fields to read from the source stream
+ * @return The initialized StreamMessageDecoder
+ */
+ private StreamMessageDecoder createMessageDecoder(Set<String> fieldsToRead) {
+ String decoderClass = _streamConfig.getDecoderClass();
+ try {
+ StreamMessageDecoder decoder = PluginManager.get().createInstance(decoderClass);
+ decoder.init(fieldsToRead, _streamConfig, _tableConfig, _schema);
+ return decoder;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Caught exception while creating StreamMessageDecoder from stream config: " + _streamConfig, e);
+ }
+ }
+
+ private class PartitionConsumer implements Runnable {
+ @Override
+ public void run() {
Review Comment:
```
// TODO:
// When reaching here, the current consuming segment has already acquired the consumer semaphore, but there is
// no guarantee that the previous consuming segment is already persisted (replaced with immutable segment). It
// can potentially cause the following problems:
// 1. The snapshot for the previous consuming segment might not be taken since it is not persisted yet
// 2. If the previous consuming segment is dropped but immutable segment is not downloaded and replaced yet,
// it might cause inconsistency (especially for partial upsert because events are not consumed in sequence)
// To address this problem, we should consider releasing the consumer semaphore after the consuming segment is
// persisted.
// Take upsert snapshot before starting consuming events
if (_partitionUpsertMetadataManager != null) {
if (_tableConfig.getUpsertMetadataTTL() > 0) {
// If upsertMetadataTTL is enabled, we will remove expired primary keys from upsertMetadata
// AFTER taking a snapshot. Taking the snapshot first is crucial to capture the final
// state of each key before it exits the TTL window. Out-of-TTL segments are skipped in
// the doAddSegment flow, and the snapshot is used to enableUpsert on the immutable out-of-TTL segment.
// If no snapshot is found, the entire segment is marked as valid and queryable.
_partitionUpsertMetadataManager.takeSnapshot();
_partitionUpsertMetadataManager.removeExpiredPrimaryKeys();
} else {
// We should remove deleted-keys first and then take a snapshot. This is because the deletedKeysTTL
// flow removes keys from the map and updates to remove valid doc IDs. By taking the snapshot immediately
// after this process, we save one commit cycle, ensuring that the deletion of valid doc IDs is reflected
// immediately
_partitionUpsertMetadataManager.removeExpiredPrimaryKeys();
_partitionUpsertMetadataManager.takeSnapshot();
}
}
```
Do we need to worry about this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941094499
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
+
+ RUNNING_JOBS.put(jobId, job);
+ doReIngestSegment(manager, segmentName, tableNameWithType, indexLoadingConfig, tableDataManager);
+ } catch (Exception e) {
+ LOGGER.error("Error during async re-ingestion for job {} (segment={})", jobId, segmentName, e);
+ } finally {
+ isIngesting.set(false);
+ RUNNING_JOBS.remove(jobId);
+ }
+ });
+
+ ReIngestionResponse immediateResponse = new ReIngestionResponse(
+ "Re-ingestion job submitted successfully with jobId: " + jobId);
+ return Response.ok(immediateResponse).build();
+ }
+
+ /**
+ * The actual re-ingestion logic, moved into a separate method for clarity.
+ * This is essentially the old synchronous logic you had in reIngestSegment.
+ */
+ private void doReIngestSegment(StatelessRealtimeSegmentDataManager manager, String segmentName,
+ String tableNameWithType, IndexLoadingConfig indexLoadingConfig, TableDataManager tableDataManager)
+ throws Exception {
+ try {
+ manager.startConsumption();
+ waitForCondition((Void) -> manager.isDoneConsuming(), CHECK_INTERVAL_MS,
+ CONSUMPTION_END_TIMEOUT_MS, 0);
+ manager.stopConsumption();
+
+ if (!manager.isSuccess()) {
+ throw new Exception("Consumer failed: " + manager.getConsumptionException());
+ }
+
+ LOGGER.info("Starting build for segment {}", segmentName);
+ StatelessRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor =
+ manager.buildSegmentInternal();
+
+ File segmentTarFile = segmentBuildDescriptor.getSegmentTarFile();
+ if (segmentTarFile == null) {
+ throw new Exception("Failed to build segment: " + segmentName);
+ }
+
+ ServerSegmentCompletionProtocolHandler protocolHandler =
+ new ServerSegmentCompletionProtocolHandler(_serverInstance.getServerMetrics(), tableNameWithType);
+
+ AuthProvider authProvider = protocolHandler.getAuthProvider();
+ List<Header> headers = AuthProviderUtils.toRequestHeaders(authProvider);
+
+ String controllerUrl = getControllerUrl(tableNameWithType, protocolHandler);
+
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ String segmentStoreUri = indexLoadingConfig.getSegmentStoreURI();
+ String destUriStr = StringUtil.join(File.separator, segmentStoreUri, rawTableName,
+ SegmentCompletionUtils.generateTmpSegmentFileName(segmentName));
+
+ try (PinotFS pinotFS = PinotFSFactory.create(new URI(segmentStoreUri).getScheme())) {
+ URI destUri = new URI(destUriStr);
+ if (pinotFS.exists(destUri)) {
+ pinotFS.delete(destUri, true);
+ }
+ pinotFS.copyFromLocalFile(segmentTarFile, destUri);
+ }
+
+ headers.add(new BasicHeader(FileUploadDownloadClient.CustomHeaders.DOWNLOAD_URI, destUriStr));
+ pushSegmentMetadata(tableNameWithType, controllerUrl, segmentTarFile, headers, segmentName, protocolHandler);
+
+ // Wait for segment to be uploaded
+ waitForCondition((Void) -> {
+ SegmentZKMetadata zkMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (zkMetadata.getStatus() != CommonConstants.Segment.Realtime.Status.UPLOADED) {
+ return false;
+ }
+ SegmentDataManager segDataManager = tableDataManager.acquireSegment(segmentName);
+ return segDataManager instanceof ImmutableSegmentDataManager;
Review Comment:
Why is this check necessary ?
Why are we waiting for the reset to succeed and the segment to get downloaded. Won't the following next reset download the segment ?
I think this is related to the check that we added to `replaceSegmentIfCrcMismatch` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1941429595
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,533 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MIN_REINGESTION_THREADS = 4;
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ new ThreadPoolExecutor(MIN_REINGESTION_THREADS, MAX_PARALLEL_REINGESTIONS, 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
+ new ThreadFactoryBuilder().setNameFormat("reingestion-worker-%d").build());
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ InstanceDataManager instanceDataManager = _serverInstance.getInstanceDataManager();
+ if (instanceDataManager == null) {
+ throw new WebApplicationException("Invalid server initialization", Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ TableDataManager tableDataManager = instanceDataManager.getTableDataManager(tableNameWithType);
+ if (tableDataManager == null) {
+ throw new WebApplicationException("Table data manager not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ IndexLoadingConfig indexLoadingConfig = tableDataManager.fetchIndexLoadingConfig();
+ LOGGER.info("Executing re-ingestion for table: {}, segment: {}", tableNameWithType, segmentName);
+
+ // Get TableConfig, Schema, ZK metadata
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ if (tableConfig == null) {
+ throw new WebApplicationException("Table config not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+ Schema schema = indexLoadingConfig.getSchema();
+ if (schema == null) {
+ throw new WebApplicationException("Schema not found for table: " + tableNameWithType,
+ Response.Status.NOT_FOUND);
+ }
+
+ SegmentZKMetadata segmentZKMetadata = tableDataManager.fetchZKMetadata(segmentName);
+ if (segmentZKMetadata == null) {
+ throw new WebApplicationException("Segment metadata not found for segment: " + segmentName,
+ Response.Status.NOT_FOUND);
+ }
+
+ // Check if download url is present
+ if (segmentZKMetadata.getDownloadUrl() != null) {
+ throw new WebApplicationException(
+ "Download URL is already present for segment: " + segmentName + ". No need to re-ingest.",
+ Response.Status.BAD_REQUEST);
+ }
+
+ // Grab start/end offsets
+ String startOffsetStr = segmentZKMetadata.getStartOffset();
+ String endOffsetStr = segmentZKMetadata.getEndOffset();
+ if (startOffsetStr == null || endOffsetStr == null) {
+ throw new WebApplicationException("Null start/end offset for segment: " + segmentName,
+ Response.Status.INTERNAL_SERVER_ERROR);
+ }
+
+ // Check if this segment is already being re-ingested
+ AtomicBoolean isIngesting = SEGMENT_INGESTION_MAP.computeIfAbsent(segmentName, k -> new AtomicBoolean(false));
+ if (!isIngesting.compareAndSet(false, true)) {
+ return Response.status(Response.Status.CONFLICT)
+ .entity("Re-ingestion for segment: " + segmentName + " is already in progress.")
+ .build();
+ }
+
+ // Generate a jobId for tracking
+ String jobId = UUID.randomUUID().toString();
+ ReIngestionJob job = new ReIngestionJob(jobId, tableNameWithType, segmentName);
+
+ // Kick off the actual work asynchronously
+ REINGESTION_EXECUTOR.submit(() -> {
+ try {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+
+ Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMaps(tableConfig).get(0);
+ StreamConfig streamConfig = new StreamConfig(tableNameWithType, streamConfigMap);
+
+ StatelessRealtimeSegmentDataManager manager = new StatelessRealtimeSegmentDataManager(
+ segmentName, tableNameWithType, partitionGroupId, segmentZKMetadata, tableConfig, schema,
+ indexLoadingConfig, streamConfig, startOffsetStr, endOffsetStr, _serverInstance.getServerMetrics());
Review Comment:
OK, for now, I am going to pass `null` here. We lose a bit of observability with it but won't be messing up realtime consumption metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on PR #14920:
URL: https://github.com/apache/pinot/pull/14920#issuecomment-2634466525
> When a segment is re-ingested, it should be `DONE` instead of `UPLOADED`. I didn't find the semaphore related logic. Is the PR description up to date?
During reingestion segment metadata upload is triggered after segment building. Hence, the segment status appears as "UPLOADED" rather than "DONE".
Upon upload completion, a segment refresh operation is triggered, which replaces the failed RealtimeSegmentDataManager with an ImmutableSegmentDataManager. During this replacement process, the system automatically releases associated semaphores.
Once the segment is refreshed, we send a reset message so that it starts showing up as ONLINE in EV
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939687054
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
Review Comment:
Not needed. removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939730030
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
This check was added in the following PR - https://github.com/apache/pinot/pull/12886/files
What problems do you see with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938852738
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
+ serverHostPort = serverHostPort.replace(HTTP + "://", "");
+ }
+
+ String serverHost = serverHostPort.split(":")[0];
+ String serverPort = serverHostPort.split(":")[1];
+
+ URI reIngestUri = getURI(scheme, serverHost, Integer.parseInt(serverPort), REINGEST_SEGMENT_PATH);
+
+ Map<String, Object> requestJson = new HashMap<>();
+ requestJson.put("tableNameWithType", tableNameWithType);
+ requestJson.put("segmentName", segmentName);
+
+ String jsonPayload = JsonUtils.objectToString(requestJson);
+ SimpleHttpResponse response =
+ HttpClient.wrapAndThrowHttpException(_httpClient.sendJsonPostRequest(reIngestUri, jsonPayload));
+
+ // Check that we got a 2xx response
+ int statusCode = response.getStatusCode();
+ if (statusCode / 100 != 2) {
+ throw new IOException(String.format("Failed POST to %s, HTTP %d: %s",
Review Comment:
The above
`HttpClient.wrapAndThrowHttpException` would throw an exception and this won't be printed in case of `status codes >= 300`.
Either we can catch it here to print the log or print it in the calling function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1938931996
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1274,6 +1275,47 @@ public File downloadUntarFileStreamed(URI uri, File dest, AuthProvider authProvi
httpHeaders, maxStreamRateInByte);
}
+ /**
+ * Invokes the server's reIngestSegment API via a POST request with JSON payload,
+ * using Simple HTTP APIs.
+ *
+ * POST http://[serverURL]/reIngestSegment
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ */
+ public void triggerReIngestion(String serverHostPort, String tableNameWithType, String segmentName)
+ throws IOException, URISyntaxException, HttpErrorStatusException {
+ String scheme = HTTP;
+ if (serverHostPort.contains(HTTPS)) {
+ scheme = HTTPS;
+ serverHostPort = serverHostPort.replace(HTTPS + "://", "");
+ } else if (serverHostPort.contains(HTTP)) {
Review Comment:
If not, we can simplify this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939084497
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2096,6 +2104,137 @@ URI createSegmentPath(String rawTableName, String segmentName) {
return URIUtils.getUri(_controllerConf.getDataDir(), rawTableName, URIUtils.encode(segmentName));
}
+ /**
+ * Re-ingests segments that are in ERROR state in EV but ONLINE in IS with no peer copy on any server. This method
+ * will call the server reIngestSegment API
+ * on one of the alive servers that are supposed to host that segment according to IdealState.
+ *
+ * API signature:
+ * POST http://[serverURL]/reIngestSegment
+ * Request body (JSON):
+ * {
+ * "tableNameWithType": [tableName],
+ * "segmentName": [segmentName]
+ * }
+ *
+ * @param tableNameWithType The table name with type, e.g. "myTable_REALTIME"
+ */
+ public void reIngestSegmentsWithErrorState(String tableNameWithType) {
+ // Step 1: Fetch the ExternalView and all segments
+ ExternalView externalView = getExternalView(tableNameWithType);
+ IdealState idealState = getIdealState(tableNameWithType);
+ Map<String, Map<String, String>> segmentToInstanceCurrentStateMap = externalView.getRecord().getMapFields();
+ Map<String, Map<String, String>> segmentToInstanceIdealStateMap = idealState.getRecord().getMapFields();
+
+ // find segments in ERROR state in externalView
+ List<String> segmentsInErrorState = new ArrayList<>();
+ for (Map.Entry<String, Map<String, String>> entry : segmentToInstanceCurrentStateMap.entrySet()) {
+ String segmentName = entry.getKey();
+ Map<String, String> instanceStateMap = entry.getValue();
+ boolean allReplicasInError = true;
+ for (String state : instanceStateMap.values()) {
+ if (!SegmentStateModel.ERROR.equals(state)) {
+ allReplicasInError = false;
+ break;
+ }
+ }
+ if (allReplicasInError) {
+ segmentsInErrorState.add(segmentName);
+ }
+ }
+
+ if (segmentsInErrorState.isEmpty()) {
+ LOGGER.info("No segments found in ERROR state for table {}", tableNameWithType);
+ return;
+ }
+
+ // filter out segments that are not ONLINE in IdealState
+ for (String segmentName : segmentsInErrorState) {
+ Map<String, String> instanceIdealStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+ boolean isOnline = true;
+ for (String state : instanceIdealStateMap.values()) {
+ if (!SegmentStateModel.ONLINE.equals(state)) {
+ isOnline = false;
+ break;
+ }
+ }
+ if (!isOnline) {
+ segmentsInErrorState.remove(segmentName);
+ }
+ }
+
+ // Step 2: For each segment, check the ZK metadata for conditions
+ for (String segmentName : segmentsInErrorState) {
+ // Skip non-LLC segments or segments missing from the ideal state altogether
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName == null || !segmentToInstanceCurrentStateMap.containsKey(segmentName)) {
+ continue;
+ }
+
+ SegmentZKMetadata segmentZKMetadata = getSegmentZKMetadata(tableNameWithType, segmentName);
+ // We only consider segments that are in COMMITTING which is indicated by having an endOffset
+ // but have a missing or placeholder download URL
+ if (segmentZKMetadata.getStatus() == Status.COMMITTING) {
+ Map<String, String> instanceStateMap = segmentToInstanceIdealStateMap.get(segmentName);
+
+ // Step 3: “No peer has that segment.” => Re-ingest from one server that is supposed to host it and is alive
+ LOGGER.info(
+ "Segment {} in table {} is COMMITTING with missing download URL and no peer copy. Triggering re-ingestion.",
+ segmentName, tableNameWithType);
+
+ // Find at least one server that should host this segment and is alive
+ String aliveServer = findAliveServerToReIngest(instanceStateMap.keySet());
+ if (aliveServer == null) {
+ LOGGER.warn("No alive server found to re-ingest segment {} in table {}", segmentName, tableNameWithType);
+ continue;
+ }
+
+ try {
+ _fileUploadDownloadClient.triggerReIngestion(aliveServer, tableNameWithType, segmentName);
+ LOGGER.info("Successfully triggered reIngestion for segment {} on server {}", segmentName, aliveServer);
+ } catch (Exception e) {
+ LOGGER.error("Failed to call reIngestSegment for segment {} on server {}", segmentName, aliveServer, e);
+ }
+ } else if (segmentZKMetadata.getStatus() == Status.UPLOADED) {
+ LOGGER.info(
+ "Segment {} in table {} is in ERROR state with download URL present. Resetting segment to ONLINE state.",
+ segmentName, tableNameWithType);
+ _helixResourceManager.resetSegment(tableNameWithType, segmentName, null);
+ }
+ }
+ }
+
+ /**
+ * Picks one 'alive' server among a set of servers that are supposed to host the segment,
+ * e.g. by checking if Helix says it is enabled or if it appears in the live instance list.
+ * This is a simple example; adapt to your environment’s definition of “alive.”
+ */
+ private String findAliveServerToReIngest(Set<String> candidateServers) {
+ // Get the current live instances from Helix
+ HelixDataAccessor helixDataAccessor = _helixManager.getHelixDataAccessor();
+ PropertyKey.Builder keyBuilder = helixDataAccessor.keyBuilder();
+ List<String> liveInstances = helixDataAccessor.getChildNames(keyBuilder.liveInstances());
+ try {
+ // This should ideally handle https scheme as well
+ BiMap<String, String> instanceToEndpointMap =
+ _helixResourceManager.getDataInstanceAdminEndpoints(candidateServers);
+
+ if (instanceToEndpointMap.isEmpty()) {
+ LOGGER.warn("No instance data admin endpoints found for servers: {}", candidateServers);
+ return null;
+ }
+
+ for (String server : candidateServers) {
+ if (liveInstances.contains(server)) {
+ return instanceToEndpointMap.get(server);
Review Comment:
The method description aims to find the server that is enabled in the Helix and is present in the live instances. This would return a null in case the segment is not present in `instanceToEndpointMap`.
We need to ensure this is non-null before returning it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939111661
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -389,15 +390,21 @@ protected void replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
IndexLoadingConfig indexLoadingConfig)
throws Exception {
String segmentName = segmentDataManager.getSegmentName();
- Preconditions.checkState(segmentDataManager instanceof ImmutableSegmentDataManager,
- "Cannot replace CONSUMING segment: %s in table: %s", segmentName, _tableNameWithType);
- SegmentMetadata localMetadata = segmentDataManager.getSegment().getSegmentMetadata();
- if (hasSameCRC(zkMetadata, localMetadata)) {
- _logger.info("Segment: {} has CRC: {} same as before, not replacing it", segmentName, localMetadata.getCrc());
- return;
+ TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+ // For pauseless tables, we should replace the segment if download url is missing even if crc is same
+ // Without this the reingestion of ERROR segments in pauseless tables fails
+ // as the segment data manager is still an instance of RealtimeSegmentDataManager
+ if (!PauselessConsumptionUtils.isPauselessEnabled(tableConfig)) {
Review Comment:
@noob-se7en Can you please review this part once. Feel free to post any questions regarding the context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]
Posted by "9aman (via GitHub)" <gi...@apache.org>.
9aman commented on code in PR #14920:
URL: https://github.com/apache/pinot/pull/14920#discussion_r1939232541
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.api.resources;
+
+import com.google.common.base.Function;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiKeyAuthDefinition;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.SecurityDefinition;
+import io.swagger.annotations.SwaggerDefinition;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.inject.Inject;
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.NameValuePair;
+import org.apache.hc.core5.http.message.BasicHeader;
+import org.apache.hc.core5.http.message.BasicNameValuePair;
+import org.apache.pinot.common.auth.AuthProviderUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.common.utils.URIUtils;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.core.data.manager.InstanceDataManager;
+import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
+import org.apache.pinot.core.data.manager.realtime.SegmentCompletionUtils;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.segment.local.data.manager.TableDataManager;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionRequest;
+import org.apache.pinot.server.api.resources.reingestion.ReIngestionResponse;
+import org.apache.pinot.server.api.resources.reingestion.utils.StatelessRealtimeSegmentDataManager;
+import org.apache.pinot.server.realtime.ControllerLeaderLocator;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.server.starter.ServerInstance;
+import org.apache.pinot.spi.auth.AuthProvider;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.StringUtil;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.spi.utils.CommonConstants.DATABASE;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
+
+
+@Api(tags = "ReIngestion", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
+ @Authorization(value = DATABASE)})
+@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
+ @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER,
+ key = SWAGGER_AUTHORIZATION_KEY,
+ description = "The format of the key is ```\"Basic <token>\" or \"Bearer <token>\"```"),
+ @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE,
+ description = "Database context passed through http header. If no context is provided 'default' database "
+ + "context will be considered.")}))
+@Path("/")
+public class ReIngestionResource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIngestionResource.class);
+
+ //TODO: Make this configurable
+ private static final int MAX_PARALLEL_REINGESTIONS = 8;
+
+ // Tracks if a particular segment is currently being re-ingested
+ private static final ConcurrentHashMap<String, AtomicBoolean>
+ SEGMENT_INGESTION_MAP = new ConcurrentHashMap<>();
+
+ // Executor for asynchronous re-ingestion
+ private static final ExecutorService REINGESTION_EXECUTOR =
+ Executors.newFixedThreadPool(MAX_PARALLEL_REINGESTIONS);
+
+ // Keep track of jobs by jobId => job info
+ private static final ConcurrentHashMap<String, ReIngestionJob> RUNNING_JOBS = new ConcurrentHashMap<>();
+ public static final long CONSUMPTION_END_TIMEOUT_MS = Duration.ofMinutes(30).toMillis();
+ public static final long UPLOAD_END_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
+ public static final long CHECK_INTERVAL_MS = Duration.ofSeconds(5).toMillis();
+
+ @Inject
+ private ServerInstance _serverInstance;
+
+ /**
+ * Simple data class to hold job details.
+ */
+ private static class ReIngestionJob {
+ private final String _jobId;
+ private final String _tableNameWithType;
+ private final String _segmentName;
+ private final long _startTimeMs;
+
+ ReIngestionJob(String jobId, String tableNameWithType, String segmentName) {
+ _jobId = jobId;
+ _tableNameWithType = tableNameWithType;
+ _segmentName = segmentName;
+ _startTimeMs = System.currentTimeMillis();
+ }
+
+ public String getJobId() {
+ return _jobId;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ public long getStartTimeMs() {
+ return _startTimeMs;
+ }
+ }
+
+ /**
+ * New API to get all running re-ingestion jobs.
+ */
+ @GET
+ @Path("/reingestSegment/jobs")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation("Get all running re-ingestion jobs along with job IDs")
+ public Response getAllRunningReingestionJobs() {
+ // Filter only the jobs still marked as running
+ List<ReIngestionJob> runningJobs = new ArrayList<>(RUNNING_JOBS.values());
+ return Response.ok(runningJobs).build();
+ }
+
+ @POST
+ @Path("/reingestSegment")
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Re-ingest segment asynchronously", notes = "Returns a jobId immediately; ingestion runs in "
+ + "background.")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success", response = ReIngestionResponse.class),
+ @ApiResponse(code = 500, message = "Internal server error", response = ErrorInfo.class)
+ })
+ public Response reIngestSegment(ReIngestionRequest request) {
+ String tableNameWithType = request.getTableNameWithType();
+ String segmentName = request.getSegmentName();
+
+ if (RUNNING_JOBS.size() >= MAX_PARALLEL_REINGESTIONS) {
+ return Response.status(Response.Status.TOO_MANY_REQUESTS)
Review Comment:
Maybe using
```
private static final ExecutorService REINGESTION_EXECUTOR =
new ThreadPoolExecutor(
MAX_PARALLEL_EXECUTIONS,
MAX_PARALLEL_EXECUTIONS, fixed size)
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<>(), // unbounded queue for the reingestion tasks
new ThreadFactoryBuilder()
.setNameFormat("reingestion-worker-%d")
.build()
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org