You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Shekharrajak (via GitHub)" <gi...@apache.org> on 2026/04/14 16:43:41 UTC
[PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Shekharrajak opened a new pull request, #19311:
URL: https://github.com/apache/druid/pull/19311
Phase 1 https://github.com/apache/druid/issues/18439.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
#### Renamed the class ...
#### Added a forbidden-apis entry ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4408686768
CI checks are failing due to one flaky test unrelated to this PR : https://github.com/apache/druid/issues/19435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4036975698
##########
docs/ingestion/kafka-share-group-ingestion.md:
##########
@@ -0,0 +1,349 @@
+---
+id: kafka-share-group-ingestion
+title: "Kafka share group ingestion"
+sidebar_label: "Kafka share group ingestion"
+description: "Queue-semantics ingestion from Apache Kafka using share groups (KIP-932). Scale consumers beyond partition count with at-least-once delivery."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+Requires Apache Kafka 4.0 or higher with share groups (KIP-932) enabled on the broker.
+:::
+
+## Overview
+
+Kafka share groups (KIP-932) let multiple consumers read from the same partition concurrently. The broker manages per-record acquisition locks and explicit acknowledgement, so consumer count is not capped by partition count, joining or leaving consumers does not pause the group, and a slow record does not block its partition.
+
+Druid's `ShareGroupIndexTask` consumes from a share group and publishes segments with at-least-once delivery: records are acknowledged only after their segments are atomically registered in the metadata store.
+
+## When to use share group ingestion
+
+| Scenario | Consumer group | Share group |
+|----------|---------------|-------------|
+| Workers needed exceed partition count | Idle workers | All workers active |
+| Elastic scaling (auto-scale events) | Rebalancing pause (30-60s) | Zero pause |
+| Per-message processing time varies | Head-of-line blocking | Independent processing |
+| Ordered processing required per partition | Yes | No (delivery order not guaranteed) |
+
+Choose share groups when throughput and elastic scaling matter more than strict per-partition ordering.
+
+## Task spec
+
+Submit a `ShareGroupIndexTask` to the Overlord. There are no start/end offsets -- the broker tracks them.
+
+```json
+{
+ "type": "index_kafka_share_group",
+ "dataSchema": {
+ "dataSource": "my_datasource",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "useSchemaDiscovery": true
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": "NONE"
+ }
+ },
+ "ioConfig": {
+ "topic": "my_topic",
+ "groupId": "druid-share-group",
+ "consumerProperties": {
+ "bootstrap.servers": "kafka-broker:9092"
+ },
+ "inputFormat": {
+ "type": "json"
+ },
+ "pollTimeout": 2000
+ },
+ "tuningConfig": {
+ "type": "KafkaTuningConfig",
+ "maxRowsPerSegment": 5000000
+ }
+}
+```
+
+## IO configuration
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `topic` | String | Yes | -- | Kafka topic to consume from. |
+| `groupId` | String | Yes | -- | Share group identifier. Multiple tasks with the same `groupId` share the workload. |
+| `consumerProperties` | Map | Yes | -- | Kafka consumer properties. Must include `bootstrap.servers`. See [Consumer property restrictions](#consumer-property-restrictions). |
+| `inputFormat` | Object | Yes | -- | Input format for parsing records (json, csv, avro, etc.). |
+| `pollTimeout` | Long | No | 2000 | Poll timeout in milliseconds. |
Review Comment:
[P3] Document the actual pollTimeout default
**Finding:** The configuration table says an omitted pollTimeout defaults to 2000 ms, but ShareGroupIndexTaskIOConfig uses KafkaSupervisorIOConfig.DEFAULT_POLL_TIMEOUT_MILLIS, which is 100 ms. Users relying on this documentation will poll twenty times more frequently than the implementation indicates in the docs.
**Suggestion:** Make the documentation match the implementation, or change the new configuration default and its test to the intended 2000 ms value.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentLockAcquireAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.TimeChunkLockAcquireAction;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final LockGranularity lockGranularity;
+ private final TaskLockType lockType;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ final boolean useTimeChunkLocks = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ );
+ this.lockGranularity = useTimeChunkLocks ? LockGranularity.TIME_CHUNK : LockGranularity.SEGMENT;
+ this.lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(this::acquireLockForRestoredSegment);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
+ );
+
+ if (!addResult.isOk()) {
+ // Throw without acknowledging so the broker redelivers this batch.
+ throw new ISE(
+ "Could not allocate segment for row with timestamp[%s]",
+ row.getTimestamp()
+ );
+ }
+
+ totalRowsIngested++;
+ midBatchPersistNeeded |= addResult.isPersistRequired();
+
+ if (!pushThresholdLogged && addResult.isPushRequired(
+ tuningConfig.getPartitionsSpec().getMaxRowsPerSegment(),
+ tuningConfig.getPartitionsSpec()
+ .getMaxTotalRowsOr(DynamicPartitionsSpec.DEFAULT_MAX_TOTAL_ROWS)
+ )) {
+ pushThresholdLogged = true;
+ log.info(
+ "Segment row threshold reached during batch (current segment[%s], rows[%d]); "
+ + "will publish at end-of-batch boundary.",
+ addResult.getSegmentIdentifier(),
+ addResult.getNumRowsInSegment()
+ );
+ }
+ }
+ }
+
+ if (midBatchPersistNeeded) {
+ driver.persist(committerSupplier.get());
+ }
+
+ driver.persist(committerSupplier.get());
+
+ final TransactionalSegmentPublisher publisher = new TransactionalSegmentPublisher()
+ {
+ @Override
+ public SegmentPublishResult publishAnnotatedSegments(
+ @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,
+ Set<DataSegment> segmentsToPush,
+ @Nullable Object commitMetadata,
+ SegmentSchemaMapping segmentSchemaMapping
+ ) throws IOException
+ {
+ if (segmentsToPush.isEmpty()) {
+ return SegmentPublishResult.ok(segmentsToPush);
+ }
+ return toolbox.getTaskActionClient().submit(
+ SegmentTransactionalAppendAction.forSegments(segmentsToPush, segmentSchemaMapping)
Review Comment:
[P2] Select the publisher for the configured lock type
**Finding:** ShareGroupIndexTask only defaults useConcurrentLocks with putIfAbsent, so a task may explicitly set it false or choose useSharedLock/taskLockType and acquire EXCLUSIVE or SHARED locks. This publisher always submits SegmentTransactionalAppendAction, whose perform method rejects every task lock that is not APPEND, so the first non-empty batch fails during publish and its records are never acknowledged for that otherwise accepted task context.
**Suggestion:** Either reject or normalize non-APPEND lock contexts for this task, or select the publish action from the actual lock type as the existing batch-task path does; add coverage for useConcurrentLocks=false.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentLockAcquireAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.TimeChunkLockAcquireAction;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final LockGranularity lockGranularity;
+ private final TaskLockType lockType;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ final boolean useTimeChunkLocks = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ );
+ this.lockGranularity = useTimeChunkLocks ? LockGranularity.TIME_CHUNK : LockGranularity.SEGMENT;
+ this.lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(this::acquireLockForRestoredSegment);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
+ );
+
+ if (!addResult.isOk()) {
+ // Throw without acknowledging so the broker redelivers this batch.
+ throw new ISE(
+ "Could not allocate segment for row with timestamp[%s]",
+ row.getTimestamp()
+ );
+ }
+
+ totalRowsIngested++;
+ midBatchPersistNeeded |= addResult.isPersistRequired();
+
+ if (!pushThresholdLogged && addResult.isPushRequired(
+ tuningConfig.getPartitionsSpec().getMaxRowsPerSegment(),
+ tuningConfig.getPartitionsSpec()
+ .getMaxTotalRowsOr(DynamicPartitionsSpec.DEFAULT_MAX_TOTAL_ROWS)
+ )) {
+ pushThresholdLogged = true;
+ log.info(
+ "Segment row threshold reached during batch (current segment[%s], rows[%d]); "
+ + "will publish at end-of-batch boundary.",
+ addResult.getSegmentIdentifier(),
+ addResult.getNumRowsInSegment()
+ );
+ }
+ }
+ }
+
+ if (midBatchPersistNeeded) {
+ driver.persist(committerSupplier.get());
+ }
+
+ driver.persist(committerSupplier.get());
+
+ final TransactionalSegmentPublisher publisher = new TransactionalSegmentPublisher()
+ {
+ @Override
+ public SegmentPublishResult publishAnnotatedSegments(
+ @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,
+ Set<DataSegment> segmentsToPush,
+ @Nullable Object commitMetadata,
+ SegmentSchemaMapping segmentSchemaMapping
+ ) throws IOException
+ {
+ if (segmentsToPush.isEmpty()) {
+ return SegmentPublishResult.ok(segmentsToPush);
+ }
+ return toolbox.getTaskActionClient().submit(
+ SegmentTransactionalAppendAction.forSegments(segmentsToPush, segmentSchemaMapping)
+ );
+ }
+
+ @Override
+ public boolean supportsEmptyPublish()
+ {
+ return false;
+ }
+ };
+
+ final SegmentsAndCommitMetadata published = driver.publish(
+ publisher,
+ committerSupplier.get(),
+ Collections.singletonList(sequenceName)
+ ).get();
+
+ final int segmentCount = published.getSegments().size();
+ if (segmentCount > 0) {
+ log.info("Published %d segments with %d total rows.", segmentCount, totalRowsIngested);
+ }
+
+ // Acknowledge only after the segment is published; any earlier failure
+ // leaves records unacknowledged so the broker redelivers them.
+ for (Map.Entry<KafkaTopicPartition, List<Long>> entry : batchOffsets.entrySet()) {
+ for (Long offset : entry.getValue()) {
+ recordSupplier.acknowledge(entry.getKey(), offset, AcknowledgeType.ACCEPT);
+ }
+ }
+
+ final Map<KafkaTopicPartition, Optional<Exception>> commitResult = recordSupplier.commitSync();
+ long commitFailures = 0L;
+ for (Map.Entry<KafkaTopicPartition, Optional<Exception>> entry : commitResult.entrySet()) {
+ if (entry.getValue().isPresent()) {
+ commitFailures++;
+ log.warn(
+ entry.getValue().get(),
+ "Commit failed for partition[%s]. Records may be redelivered.",
+ entry.getKey()
+ );
+ }
+ }
+ if (commitFailures > 0) {
+ task.emitMetric(toolbox.getEmitter(), METRIC_COMMIT_FAILURES, commitFailures);
+ }
+ }
+
+ driver.persist(committerSupplier.get());
+ log.info("Share group ingestion complete. Total rows ingested: %d", totalRowsIngested);
+ appenderator.close();
+ appenderatorClosedNormally = true;
+ }
+ finally {
+ if (!appenderatorClosedNormally) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator; continuing teardown.");
+ }
+ }
+ }
+
+ return TaskStatus.success(task.getId());
+ }
+
+ private boolean acquireLockForRestoredSegment(SegmentIdWithShardSpec segmentId)
+ {
+ try {
+ if (lockGranularity == LockGranularity.SEGMENT) {
+ return toolbox.getTaskActionClient().submit(
+ new SegmentLockAcquireAction(
+ lockType,
Review Comment:
[P2] Avoid APPEND locks with segment granularity
**Finding:** When forceTimeChunkLock is false, this task still defaults useConcurrentLocks to true, making lockType APPEND. The segment restore branch passes that type to SegmentLockAcquireAction even though Druid permits APPEND only with time-chunk locking and TaskLockbox rejects an APPEND segment request when conflicting locks exist; SegmentAllocateAction receives the same unsupported pairing. A segment-granularity task can therefore fail to restore a persisted segment, whereas the existing SeekableStreamIndexTaskRunner uses EXCLUSIVE for this branch.
**Suggestion:** Use a supported lock-type/granularity pairing for segment mode and keep allocation, restoration, and publishing consistent, or explicitly reject forceTimeChunkLock=false; cover the configuration with a lockbox-backed test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4411685392
Struggling with CI check related to license :
```
Unsupported license: MPL 1.1
For:org.javassist javassist in: druid-kerberos
org.locationtech.jts jts-core in: parquet-extensions with: Eclipse Public License 2.0 ignoring Eclipse Distribution License - v 1.0
Error: Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:394)
at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:292)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
[INFO] simple-client-sslcontext ........................... SUCCESS [ 5.195 s]
[INFO] druid-basic-security ............................... SUCCESS [ 9.326 s]
[INFO] druid-google-extensions ............................ SUCCESS [ 7.697 s]
[INFO] druid-catalog ...................................... SUCCESS [ 9.519 s]
[INFO] druid-testing-tools ................................ SUCCESS [ 8.122 s]
[INFO] druid-testcontainers ............................... SUCCESS [ 2.149 s]
[INFO] druid-influx-extensions ............................ SUCCESS [ 4.579 s]
[INFO] druid-cassandra-storage ............................ SUCCESS [ 3.987 s]
[INFO] dropwizard-emitter ................................. SUCCESS [ 6.528 s]
[INFO] druid-cloudfiles-extensions ........................ SUCCESS [ 5.857 s]
[INFO] druid-consul-extensions ............................ SUCCESS [ 6.595 s]
[INFO] graphite-emitter ................................... SUCCESS [ 6.085 s]
[INFO] druid-distinctcount ................................ SUCCESS [ 4.219 s]
[INFO] druid-exact-count-bitmap ........................... SUCCESS [ 7.275 s]
[INFO] statsd-emitter ..................................... SUCCESS [ 6.282 s]
[INFO] druid-time-min-max ................................. SUCCESS [ 5.725 s]
[INFO] druid-virtual-columns .............................. SUCCESS [ 4.011 s]
[INFO] druid-thrift-extensions ............................ SUCCESS [ 4.915 s]
[INFO] ambari-metrics-emitter ............................. SUCCESS [ 6.306 s]
[INFO] sqlserver-metadata-storage ......................... SUCCESS [ 6.266 s]
[INFO] kafka-emitter ...................................... SUCCESS [ 6.186 s]
[INFO] druid-redis-cache .................................. SUCCESS [ 7.346 s]
[INFO] druid-opentsdb-emitter ............................. SUCCESS [ 5.524 s]
[INFO] druid-momentsketch ................................. SUCCESS [ 6.063 s]
[INFO] druid-moving-average-query ......................... SUCCESS [ 7.560 s]
[INFO] tdigestsketch ...................................... SUCCESS [ 7.308 s]
[INFO] druid-ddsketch ..................................... SUCCESS [ 7.400 s]
[INFO] druid-influxdb-emitter ............................. SUCCESS [ 4.017 s]
[INFO] gce-extensions ..................................... SUCCESS [ 6.621 s]
[INFO] aliyun-oss-extensions .............................. SUCCESS [ 7.980 s]
[INFO] prometheus-emitter ................................. SUCCESS [ 5.877 s]
[INFO] opentelemetry-emitter .............................. SUCCESS [ 8.167 s]
[INFO] grpc-query ......................................... SUCCESS [ 14.755 s]
[INFO] druid-iceberg-extensions ........................... SUCCESS [ 9.575 s]
[INFO] druid-deltalake-extensions ......................... SUCCESS [ 6.163 s]
[INFO] druid-spectator-histogram .......................... SUCCESS [ 8.500 s]
[INFO] druid-rabbit-indexing-service ...................... SUCCESS [ 7.905 s]
[INFO] druid-ranger-security .............................. SUCCESS [ 9.363 s]
[INFO] distribution ....................................... FAILURE [05:29 min]
[INFO] druid-quidem-ut .................................... SKIPPED
[INFO] druid-embedded-tests ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22:30 min
[INFO] Finished at: 2026-05-09T05:59:40Z
[INFO] ------------------------------------------------------------------------
Error: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.5.1:exec (check-licenses) on project distribution: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
Error:
Error: To see the full stack trace of the errors, re-run Maven with the -e switch.
Error: Re-run Maven using the -X switch to enable full debug logging.
Error:
Error: For more information about the errors and possible solutions, please read the following articles:
Error: [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error:
Error: After correcting the problems, you can resume the build with the command
Error: mvn <args> -rf :distribution
Error: Process completed with exit code 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3254667382
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assert.assertNotNull(polled2);
+ Assert.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(1, result.size());
+ Assert.assertTrue(result.get(0).getData().isEmpty());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assert.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+ }
+
+ @Test
+ public void testCommitSync()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.empty());
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Map.Entry<KafkaTopicPartition, Optional<Exception>> entry = result.entrySet().iterator().next();
+ Assert.assertEquals(Optional.of("test-topic"), entry.getKey().topic());
+ Assert.assertEquals(0, entry.getKey().partition());
+ Assert.assertFalse(entry.getValue().isPresent());
+ }
+
+ @Test
+ public void testCommitSyncWithError()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final KafkaException error = new KafkaException("commit failed");
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.of(error));
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Optional<Exception> maybeError = result.values().iterator().next();
+ Assert.assertTrue(maybeError.isPresent());
+ Assert.assertEquals("commit failed", maybeError.get().getMessage());
+ }
+
+ @Test
+ public void testCloseIsIdempotent()
+ {
+ supplier.close();
+ supplier.close();
+ verify(mockConsumer, Mockito.times(1)).close();
+ }
+
+ @Test
+ public void testWakeupForwardsToConsumer()
+ {
+ supplier.wakeup();
+ Mockito.verify(mockConsumer).wakeup();
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsForwardsToConsumer()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.of(45_000));
+ final Optional<Integer> lockMs = supplier.acquisitionLockTimeoutMs();
+ Assert.assertTrue(lockMs.isPresent());
+ Assert.assertEquals(Integer.valueOf(45_000), lockMs.get());
+ }
+
+ @Test
+ public void testAcknowledgeTwoArgOverloadDefaultsToAccept() throws Exception
+ {
+ final String topic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ topic, 0, 100L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(topic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11237)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4015973818
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/common/AcknowledgingRecordSupplier.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.seekablestream.common;
+
+import org.apache.druid.data.input.impl.ByteEntity;
+
+import javax.validation.constraints.NotNull;
Review Comment:
[P1] Use Jakarta validation imports
**Finding:** This new interface imports `javax.validation.constraints.NotNull`, but the current indexing-service declares only `jakarta.validation-api` and the existing `RecordSupplier` uses the Jakarta package. The same invalid import is present in `KafkaShareGroupRecordSupplier.java`; neither source can compile because the javax validation package is unavailable.
**Suggestion:** Change both new `NotNull` imports to the repository's `jakarta.validation.constraints.NotNull` package.
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
Review Comment:
[P1] Keep share-group tests on the configured JUnit platform
**Finding:** This and the other new share-group unit tests import JUnit 4 (`org.junit.*`), but the current `kafka-indexing-service` and `indexing-service` POMs declare only JUnit Jupiter; the current dependency update also removed `junit:junit` and the Vintage engine. The six new unit-test sources therefore cannot compile or be discovered in the module's current test setup.
**Suggestion:** Migrate all six new unit tests to JUnit Jupiter, or explicitly restore JUnit 4 plus the Vintage engine for these modules.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(segmentId -> true);
Review Comment:
[P2] Reacquire locks for restored segments
**Finding:** `StreamAppenderatorDriver.startJob` calls its lock helper for every restored non-pushed segment, but this lambda returns true without acquiring or validating any lock. On a same-worker retry with persisted `persist/` state, the subsequent `SegmentTransactionalAppendAction` lock-coverage check can reject those restored segments, leaving their records unacknowledged and making recovery repeatedly fail.
**Suggestion:** Pass a lock helper that reacquires the configured segment/time-chunk lock, as `SeekableStreamIndexTaskRunner` does, and cover the restored-segment retry path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4461854661
CI Check error - unrelated timeouts :
```
embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class:282 | Timed out waiting for event after [20,000]ms 20.391s
embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class:337 | Timed out waiting for event after [20,000]ms 20.015s
embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class:166 | Timed out waiting for event after [20,000]ms 20.019s
ℹ️ Posting with conclusion 'failure' to https://github.com/apache/druid/pull/19311 (sha: 2d094d5fbe6e11b6d22b1df8bf54c8cf7b7d03d2)
🚀 Publish results
ℹ️ - Test Report - test-jdk25-[I*,A*,U*] - 18604 tests run, 17847 passed, 754 skipped, 3 failed.
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class | Timed out waiting for event after [20,000]ms
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class | Timed out waiting for event after [20,000]ms
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.class | Timed out waiting for event after [20,000]ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4535152235
One CI check failing - looks like network error. Please help in re-trigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4404128722
Spending time on Integeration testing embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedShareGroupIngestionTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3212857450
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTask.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.common.config.TaskConfig;
+import org.apache.druid.indexing.common.task.AbstractTask;
+import org.apache.druid.indexing.common.task.PendingSegmentAllocatingTask;
+import org.apache.druid.indexing.common.task.TaskResource;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.ResourceAction;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Indexing task that consumes from a Kafka topic with share-group semantics
+ * (KIP-932). The broker tracks delivery state; the task acknowledges records
+ * after the segments containing them are published. Phase 1: single-threaded,
+ * no supervisor, no dedup cache.
+ */
+public class ShareGroupIndexTask extends AbstractTask implements PendingSegmentAllocatingTask
Review Comment:
Thanks for sharing, this is important for task scheduling order in TaskQueue. Updating..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3207039264
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedShareGroupIngestionTest.java:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.JsonInputFormat;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+/** End-to-end IT for Kafka share-group ingestion; requires Docker. */
+public class EmbeddedShareGroupIngestionTest extends EmbeddedClusterTestBase
+{
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private static final InputFormat DEFAULT_CSV_FORMAT =
+ new CsvInputFormat(List.of(COL_TIMESTAMP, COL_ITEM, COL_VALUE), null, null, false, 0, false);
+
+ /** Lets a freshly submitted share-group task reach STABLE before records are produced. */
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+
+ return cluster;
+ }
+
+ @Test
+ public void test_shareGroupIngestion_basicEndToEnd() throws InterruptedException
Review Comment:
Subscribe to topic with KafkaShareConsumer (groupId), poll → parse (CSV) → persist → publish → acknowledge(ACCEPT) → commitSync → broker offset persistence. SQL COUNT(*) == 10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4461903070
Another flaky test: https://github.com/apache/druid/issues/19464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3189390463
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Thanks for confirming. I still think this needs to be resolved or explicitly scoped out before this PR lands: the current task/docs already state the no-data-loss invariant, but `parseRow()` still consumes only the first row from the reader and the batch ACK then accepts the whole Kafka record. Any input format that expands one Kafka record into multiple rows would still silently lose the later rows after publish/commit. Could we either iterate and add every row before ACKing, or enforce/document a one-row-per-record limitation for this phase so the invariant is accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3087490108
##########
docs/ingestion/kafka-share-group-ingestion.md:
##########
@@ -0,0 +1,260 @@
+---
+id: kafka-share-group-ingestion
+title: "Kafka share group ingestion"
+sidebar_label: "Kafka share group ingestion"
+description: "Queue-semantics ingestion from Apache Kafka using share groups (KIP-932). Scale consumers beyond partition count with at-least-once delivery."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+Requires Apache Kafka 4.0 or higher. Share groups (KIP-932) must be enabled on the broker.
+:::
+
+## Overview
+
+Standard Kafka consumer groups bind each partition to exactly one consumer. This creates a hard scaling ceiling: you cannot have more consumers than partitions. Rebalancing when consumers join or leave pauses all consumers in the group. A single slow message blocks all subsequent messages in its partition.
+
+Kafka share groups (KIP-932) eliminate these constraints. The broker manages record delivery across consumers with per-record acquisition locks and explicit acknowledgement. Multiple consumers can read from the same partition concurrently. There is no rebalancing pause. Slow records do not block other records.
+
+Druid's share group ingestion uses `ShareGroupIndexTask` to consume from Kafka share groups and publish segments with at-least-once delivery guarantees. Records are acknowledged only after segments are atomically registered in the metadata store.
+
+## When to use share group ingestion
+
+| Scenario | Consumer group | Share group |
+|----------|---------------|-------------|
+| Workers needed exceed partition count | Idle workers | All workers active |
+| Elastic scaling (auto-scale events) | Rebalancing pause (30-60s) | Zero pause |
+| Per-message processing time varies | Head-of-line blocking | Independent processing |
+| Ordered processing required per partition | Yes | No (delivery order not guaranteed) |
+
+Use share group ingestion when throughput and elastic scaling matter more than strict per-partition ordering.
+
+## Task spec
+
+Submit a `ShareGroupIndexTask` to the Overlord. Unlike standard Kafka ingestion, there are no start/end offsets -- the broker manages offset tracking.
+
+```json
+{
+ "type": "index_kafka_share_group",
+ "dataSchema": {
+ "dataSource": "my_datasource",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "useSchemaDiscovery": true
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": "NONE"
+ }
+ },
+ "ioConfig": {
+ "topic": "my_topic",
+ "groupId": "druid-share-group",
+ "consumerProperties": {
+ "bootstrap.servers": "kafka-broker:9092"
+ },
+ "inputFormat": {
+ "type": "json"
+ },
+ "pollTimeout": 2000
+ },
+ "tuningConfig": {
+ "type": "KafkaTuningConfig",
+ "maxRowsPerSegment": 5000000
+ }
+}
+```
+
+## IO configuration
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `topic` | String | Yes | -- | Kafka topic to consume from. |
+| `groupId` | String | Yes | -- | Share group identifier. Multiple tasks with the same `groupId` share the workload. |
+| `consumerProperties` | Map | Yes | -- | Kafka consumer properties. Must include `bootstrap.servers`. |
+| `inputFormat` | Object | Yes | -- | Input format for parsing records (json, csv, avro, etc.). |
+| `pollTimeout` | Long | No | 2000 | Poll timeout in milliseconds. |
+
+## How it works
+
+1. The task subscribes to the topic using a `KafkaShareConsumer` with the configured `groupId`.
+2. The broker delivers batches of records with acquisition locks.
+3. The task parses records, adds rows to an appenderator, and persists segments.
+4. Segments are published atomically to the metadata store.
+5. After successful publish, the task acknowledges all records in the batch with `ACCEPT`.
+6. The task calls `commitSync()` to commit acknowledgements to the broker.
+7. On task failure, unacknowledged records are redelivered by the broker to another consumer in the share group.
+
+## Safety invariants
+
+1. Records are acknowledged with `ACCEPT` only after the segment containing them is atomically registered in the metadata store. No data loss on task failure.
+2. Every polled record reaches exactly one terminal state: `ACCEPT` (processed), `RELEASE` (redelivered), or task crash (broker redelivers after lock timeout).
+
+## Scaling
+
+Multiple tasks with the same `groupId` distribute the workload automatically. Unlike consumer groups, you can run more tasks than partitions:
+
+```
+Topic: 4 partitions
+Tasks with same groupId: 20
+Result: All 20 tasks actively consuming (broker distributes records)
+```
+
+Adding or removing tasks does not trigger a rebalancing pause. New tasks begin consuming immediately.
+
+## Delivery semantics
+
+Share group ingestion provides **at-least-once** delivery. On task failure, records between the last committed acknowledgement and the failure point are redelivered. Duplicate records may be ingested across task restarts. A deduplication cache is planned for a future release.
+
+## Limitations (current release)
+
+- Single-threaded ingestion per task. Two-thread architecture with background lock renewal is planned.
+- No supervisor integration. Tasks must be submitted manually via the Overlord API.
+- No deduplication cache. Redelivered records after task failure may produce duplicates.
+- Delivery order within a partition is not guaranteed.
+
+## Demo: end-to-end validation with Druid UI
+
+### Prerequisites
+
+- Java 17
+- Kafka 4.2.0 (with share groups enabled)
+- Druid 31.0.0 release (downloaded)
+
+### Step 1: Start Kafka with share groups
+
+```bash
+cd kafka_2.13-4.2.0
+
+KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
+bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
+
+# Enable share groups
+echo "group.share.enable=true" >> config/server.properties
+echo "group.share.record.lock.duration.ms=30000" >> config/server.properties
+
+bin/kafka-server-start.sh config/server.properties
+```
+
+### Step 2: Create topic and produce messages
+
+```bash
+cd kafka_2.13-4.2.0
+
+bin/kafka-topics.sh --create --topic druid-share-test --partitions 4 --bootstrap-server localhost:9092
+
+bin/kafka-console-producer.sh --topic druid-share-test --bootstrap-server localhost:9092
+```
+
+Paste these JSON records:
+
+```json
+{"__time":"2025-06-01T00:00:00.000Z","item":"widget_a","value":100,"category":"electronics"}
+{"__time":"2025-06-01T01:00:00.000Z","item":"widget_b","value":250,"category":"clothing"}
+{"__time":"2025-06-01T02:00:00.000Z","item":"widget_c","value":50,"category":"electronics"}
+{"__time":"2025-06-01T03:00:00.000Z","item":"widget_d","value":175,"category":"food"}
+{"__time":"2025-06-01T04:00:00.000Z","item":"widget_e","value":320,"category":"electronics"}
+```
+
+### Step 3: Build the extension and set up Druid
+
+```bash
+# Build the kafka-indexing-service extension
+cd /path/to/druid
+JAVA_HOME=$(/usr/libexec/java_home -v 17) mvn package \
+ -pl extensions-core/kafka-indexing-service -am \
+ -Pskip-static-checks -Dmaven.test.skip=true -T1C -q
+
+# Download and extract Druid release
+cd ~/Downloads
+curl -O https://dlcdn.apache.org/druid/31.0.0/apache-druid-31.0.0-bin.tar.gz
+tar -xzf apache-druid-31.0.0-bin.tar.gz
+cd apache-druid-31.0.0
+
+# Replace the kafka extension with our build
+rm extensions/druid-kafka-indexing-service/*.jar
Review Comment:
We can build from the source and use the tar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4404554172
To make sure no data loss we are ack only after the publish completed https://github.com/apache/druid/pull/19311/changes#diff-487176908fb14d87486db6f683c9ddb34465f3a1572f5c01b11d7b9c6a482289R381
Although this will be slow and no concurrency but I think for the intial phase this is fine and we can improve the performance step by step.
WDYT @gianm @a2l007 @clintropolis @FrankChen021 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3283388876
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assert.assertNotNull(polled2);
+ Assert.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(1, result.size());
+ Assert.assertTrue(result.get(0).getData().isEmpty());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assert.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+ }
+
+ @Test
+ public void testCommitSync()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.empty());
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Map.Entry<KafkaTopicPartition, Optional<Exception>> entry = result.entrySet().iterator().next();
+ Assert.assertEquals(Optional.of("test-topic"), entry.getKey().topic());
+ Assert.assertEquals(0, entry.getKey().partition());
+ Assert.assertFalse(entry.getValue().isPresent());
+ }
+
+ @Test
+ public void testCommitSyncWithError()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final KafkaException error = new KafkaException("commit failed");
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.of(error));
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Optional<Exception> maybeError = result.values().iterator().next();
+ Assert.assertTrue(maybeError.isPresent());
+ Assert.assertEquals("commit failed", maybeError.get().getMessage());
+ }
+
+ @Test
+ public void testCloseIsIdempotent()
+ {
+ supplier.close();
+ supplier.close();
+ verify(mockConsumer, Mockito.times(1)).close();
+ }
+
+ @Test
+ public void testWakeupForwardsToConsumer()
+ {
+ supplier.wakeup();
+ Mockito.verify(mockConsumer).wakeup();
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsForwardsToConsumer()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.of(45_000));
+ final Optional<Integer> lockMs = supplier.acquisitionLockTimeoutMs();
+ Assert.assertTrue(lockMs.isPresent());
+ Assert.assertEquals(Integer.valueOf(45_000), lockMs.get());
+ }
+
+ @Test
+ public void testAcknowledgeTwoArgOverloadDefaultsToAccept() throws Exception
+ {
+ final String topic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ topic,
+ 0,
+ 100L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(topic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, topic, 0), 100L);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testGetPartitionIdsReturnsEmptySet()
+ {
+ Assert.assertEquals(Set.of(), supplier.getPartitionIds("any-stream"));
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsEmpty()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.empty());
+ Assert.assertTrue(supplier.acquisitionLockTimeoutMs().isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeMap_multiPartition_acknowledgesAllRecords()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> r0 = new ConsumerRecord<>(testTopic, 0, 10L,
+ "k0".getBytes(StandardCharsets.UTF_8), "v0".getBytes(StandardCharsets.UTF_8));
+ final ConsumerRecord<byte[], byte[]> r1 = new ConsumerRecord<>(testTopic, 1, 20L,
+ "k1".getBytes(StandardCharsets.UTF_8), "v1".getBytes(StandardCharsets.UTF_8));
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(r0));
+ recordMap.put(new TopicPartition(testTopic, 1), List.of(r1));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11249)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3298099715
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,497 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(segmentId -> true);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ SEQUENCE_NAME,
Review Comment:
Thanks for checking .
FIx : https://github.com/apache/druid/pull/19311/changes/bf4b2ec9a66b2c67566f9a76e44e4be6758a20b9
tests: https://github.com/apache/druid/pull/19311/changes/cd7873b72143961e30e8749d02e4bf02a15aabde
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3208936088
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Thanks, the current version now uses `StreamChunkReader` and iterates every returned `InputRow` before acknowledging the batch, so the multi-row and zero-row cases from this thread look addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3259677059
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupPartitionRebalancingIT.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Verifies that the Kafka broker automatically rebalances new partitions to share-group consumers
+ * without any Druid-side intervention (no supervisor, no task restart).
Review Comment:
without any Druid-side intervention (no supervisor, no task restart) - Main advantages of using share group - no stop the world case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3141459226
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
[P1] Share-group ingestion drops rows when one Kafka record expands to multiple rows
This runner processes each Kafka message as though it produces exactly one InputRow. `parseRow()` only returns `iterator.next()`, so later rows from the same message are silently discarded, and zero-row messages can pass null into `driver.add`. Because the code still ACKs the Kafka record after publish, those skipped rows are lost permanently rather than being retried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3276482957
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexerCrashIT.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Verifies at-least-once semantics: uncommitted records are redelivered after Indexer crash.
+ */
+public class ShareGroupIndexerCrashIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+ private static final String GROUP_ID = "crash-it-group";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(60)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_indexerCrash_midIngestion_noDataLossAfterRecovery() throws Exception
+ {
+ final String topic = dataSource + "_crash_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset(GROUP_ID, "earliest");
+
+ // Produce all records before starting task so Indexer is polling them when it crashes.
+ final int batchA = 20;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(batchA, 0, "2025-09-01"));
+
+ // Task 1: starts ingesting but Indexer crashes before it completes.
+ final String taskId1 = submitTask(topic);
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Wait for at least some rows to be processed (but not all published to segments).
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(1)
+ );
+
+ // Simulate JVM crash: stop the indexer abruptly (no graceful shutdown).
+ indexer.stop();
+
+ // Restart the indexer.
+ indexer.start();
+
+ // Task 2: same group-id. Broker redelivers any unacknowledged records.
+ final String taskId2 = submitTask(topic);
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Produce a second batch to ensure Task 2 is actively consuming.
+ final int batchB = 10;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(batchB, batchA, "2025-09-02"));
+
+ // Give Task 2 time to consume redelivered + new records.
+ // Post-restart emitter has a fresh counter, so we use a dwell + SQL check instead.
+ Thread.sleep(15_000L);
+
+ cluster.callApi().onLeaderOverlord(o -> o.cancelTask(taskId2));
+ cluster.callApi().waitForTaskToFinish(taskId2, overlord.latchableEmitter());
+
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ // At-least-once guarantee: row count >= batchA + batchB (no loss); may be slightly higher due to redelivery.
+ final long rowCount = Long.parseLong(cluster.runSql("SELECT COUNT(*) FROM %s", dataSource));
+ Assertions.assertTrue(
+ rowCount >= batchA + batchB,
+ "Expected at least [" + (batchA + batchB) + "] rows but got [" + rowCount + "]"
+ );
+ }
+
+ @Test
+ public void test_indexerCrash_beforeAnyAck_allRecordsRedelivered() throws Exception
+ {
+ final String topic = dataSource + "_crash_no_ack_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset(GROUP_ID + "-noack", "earliest");
+
+ final int numRecords = 10;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(numRecords, 0, "2025-10-01"));
+
+ // Task 1: starts but is killed before its first publish (no ack committed).
+ submitTask(topic, GROUP_ID + "-noack");
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ indexer.stop();
+ indexer.start();
+
+ // Task 2 (same group): broker must redeliver all numRecords because none were acked.
+ final String taskId2 = submitTask(topic, GROUP_ID + "-noack");
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Give Task 2 time to consume redelivered records before cancel.
+ Thread.sleep(15_000L);
Review Comment:
Updated https://github.com/apache/druid/pull/19311/changes/94cbff5b73dfb8bf88101cd26ed4fe431b41f6ea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4498305277
fixing one unit test :
```
mbedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.class:96 | org.apache.druid.rpc.HttpResponseException: Server error [400 Bad Request]; body: {"error":"Please make sure to load all the necessary extensions and jars with type 'ShareGroupIndexTask'. Could not resolve type id 'ShareGroupIndexTask' as a subtype of `org.apache.druid.indexing.common.task.Task` known type ids = [archive, compact, index, index_hadoop, index_kafka, index_kafka_share_group, index_parallel, index_sub, kill, move, noop, partial_dimension_cardinality, partial_dimension_distribution, partial_index_generate, partial_index_generic_merge, partial_range_index_generate, query_controller, query_worker, restore, single_phase_sub_task]"} 0.153s
ℹ️ Posting with conclusion 'failure' to https://github.com/apache/druid/pull/19311 (sha: ad1f11ac54ba82b18a9bee9c1d37e4a853b2e36b)
🚀 Publish results
ℹ️ - Test Report - test-jdk25-[S*] - 38737 tests run, 38719 passed, 13 skipped, 5 failed.
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/ShareGroupPartitionRebalancingIT.class | Expected at least [20] rows but got [10] ==> expected: <true> but was: <false>
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/ShareGroupPartitionRebalancingIT.class | Expected at least [30] rows but got [10] ==> expected: <true> but was: <false>
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/ShareGroupIndexerCrashIT.class | Timed out waiting for event after [60,000]ms
🧪 - embedded-tests/target/test-classes/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.class | org.apache.druid.rpc.HttpResponseException: Server error [400 Bad Request]; body: {"error":"Please make sure to load all the necessary extensions and jars with type 'ShareGroupIndexTask'. Could not resolve type id 'ShareGroupIndexTask' as a subtype of `org.apache.druid.indexing.common.task.Task` known type ids = [archive, compact, index, index_hadoop, index_kafka, index_kafka_share_group, index_parallel, index_sub, kill, move, noop, partial_dimension_cardinality, partial_dimension_distribution, partial_index_generate, partial_index_generic_merge, partial_range_index_generate, query_controller, query_worker, restore, single_phase_sub_task]"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4474476759
CI checks are looking fine.
Almost all IT tests are written here embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedShareGroupIngestionTest.java
Let me see if we can have more IT tests so that it is easy to understand the how to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3212551979
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTask.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.common.config.TaskConfig;
+import org.apache.druid.indexing.common.task.AbstractTask;
+import org.apache.druid.indexing.common.task.PendingSegmentAllocatingTask;
+import org.apache.druid.indexing.common.task.TaskResource;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.ResourceAction;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Indexing task that consumes from a Kafka topic with share-group semantics
+ * (KIP-932). The broker tracks delivery state; the task acknowledges records
+ * after the segments containing them are published. Phase 1: single-threaded,
+ * no supervisor, no dedup cache.
+ */
+public class ShareGroupIndexTask extends AbstractTask implements PendingSegmentAllocatingTask
Review Comment:
[P2] Share-group tasks inherit priority 0
Because ShareGroupIndexTask extends AbstractTask directly and does not override getPriority(), tasks submitted without an explicit context priority use Tasks.DEFAULT_TASK_PRIORITY (0). Existing Kafka realtime tasks use Tasks.DEFAULT_REALTIME_TASK_PRIORITY (75), and TaskLockbox uses task.getPriority() when acquiring or competing for append locks, so these ingestion tasks can lose lock contention or be ordered differently from equivalent Kafka ingestion. Add the same realtime priority override used by SeekableStreamIndexTask.
##########
extensions-core/kafka-indexing-service/pom.xml:
##########
@@ -252,6 +257,30 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <!--
+ The bundled in-JVM test broker (kafka_2.13:3.9.x, ZooKeeper-mode) was
+ compiled against kafka-clients 3.9.x and references Utils.mkSet(Object[]),
+ which was removed in kafka-clients 4.x. This module pins kafka-clients to
+ 4.2.0 to enable KIP-932 share-group support, so KafkaConfig static
+ initialization throws NoSuchMethodError under any JVM. The tests below
+ depend on TestBroker and must be skipped until the test broker migrates to
+ kafka_2.13:4.x (KRaft) or Testcontainers. Share-group code paths are
+ covered by ShareGroupIndexTaskRunnerTest, KafkaShareGroupRecordSupplierTest
+ (mock-based) and embedded-tests/EmbeddedShareGroupIngestionTest
+ (Testcontainers). Tracked as a follow-up of apache/druid#19311.
+ -->
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <exclude>**/KafkaIndexTaskTest.java</exclude>
Review Comment:
[P2] Kafka compatibility tests are skipped globally
This Surefire exclusion removes the main KafkaIndexTask, KafkaRecordSupplier, KafkaSamplerSpec, and KafkaSupervisor test classes from every normal test run. Since this PR also upgrades kafka-clients to 4.2.0, those are the regression tests most likely to catch compatibility breaks in existing index_kafka ingestion; the new share-group mock and embedded tests do not cover that existing API surface. Please migrate or narrowly isolate the TestBroker-dependent cases instead of excluding the whole classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-5891721772
We will break this into pieces but these are rough code that works for IT tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4111419687
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentLockAcquireAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.TimeChunkLockAcquireAction;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final LockGranularity lockGranularity;
+ private final TaskLockType lockType;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ final boolean useTimeChunkLocks = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ );
+ this.lockGranularity = useTimeChunkLocks ? LockGranularity.TIME_CHUNK : LockGranularity.SEGMENT;
+ this.lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(this::acquireLockForRestoredSegment);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
+ );
+
+ if (!addResult.isOk()) {
+ // Throw without acknowledging so the broker redelivers this batch.
+ throw new ISE(
+ "Could not allocate segment for row with timestamp[%s]",
+ row.getTimestamp()
+ );
+ }
+
+ totalRowsIngested++;
+ midBatchPersistNeeded |= addResult.isPersistRequired();
+
+ if (!pushThresholdLogged && addResult.isPushRequired(
+ tuningConfig.getPartitionsSpec().getMaxRowsPerSegment(),
+ tuningConfig.getPartitionsSpec()
+ .getMaxTotalRowsOr(DynamicPartitionsSpec.DEFAULT_MAX_TOTAL_ROWS)
+ )) {
+ pushThresholdLogged = true;
+ log.info(
+ "Segment row threshold reached during batch (current segment[%s], rows[%d]); "
+ + "will publish at end-of-batch boundary.",
+ addResult.getSegmentIdentifier(),
+ addResult.getNumRowsInSegment()
+ );
+ }
+ }
+ }
+
+ if (midBatchPersistNeeded) {
+ driver.persist(committerSupplier.get());
+ }
+
+ driver.persist(committerSupplier.get());
+
+ final TransactionalSegmentPublisher publisher = new TransactionalSegmentPublisher()
+ {
+ @Override
+ public SegmentPublishResult publishAnnotatedSegments(
+ @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,
+ Set<DataSegment> segmentsToPush,
+ @Nullable Object commitMetadata,
+ SegmentSchemaMapping segmentSchemaMapping
+ ) throws IOException
+ {
+ if (segmentsToPush.isEmpty()) {
+ return SegmentPublishResult.ok(segmentsToPush);
+ }
+ return toolbox.getTaskActionClient().submit(
+ SegmentTransactionalAppendAction.forSegments(segmentsToPush, segmentSchemaMapping)
+ );
+ }
+
+ @Override
+ public boolean supportsEmptyPublish()
+ {
+ return false;
+ }
+ };
+
+ final SegmentsAndCommitMetadata published = driver.publish(
+ publisher,
+ committerSupplier.get(),
+ Collections.singletonList(sequenceName)
+ ).get();
+
+ final int segmentCount = published.getSegments().size();
+ if (segmentCount > 0) {
+ log.info("Published %d segments with %d total rows.", segmentCount, totalRowsIngested);
+ }
+
+ // Acknowledge only after the segment is published; any earlier failure
+ // leaves records unacknowledged so the broker redelivers them.
+ for (Map.Entry<KafkaTopicPartition, List<Long>> entry : batchOffsets.entrySet()) {
+ for (Long offset : entry.getValue()) {
+ recordSupplier.acknowledge(entry.getKey(), offset, AcknowledgeType.ACCEPT);
+ }
+ }
+
+ final Map<KafkaTopicPartition, Optional<Exception>> commitResult = recordSupplier.commitSync();
+ long commitFailures = 0L;
+ for (Map.Entry<KafkaTopicPartition, Optional<Exception>> entry : commitResult.entrySet()) {
+ if (entry.getValue().isPresent()) {
+ commitFailures++;
+ log.warn(
+ entry.getValue().get(),
+ "Commit failed for partition[%s]. Records may be redelivered.",
+ entry.getKey()
+ );
+ }
+ }
+ if (commitFailures > 0) {
+ task.emitMetric(toolbox.getEmitter(), METRIC_COMMIT_FAILURES, commitFailures);
+ }
+ }
+
+ driver.persist(committerSupplier.get());
+ log.info("Share group ingestion complete. Total rows ingested: %d", totalRowsIngested);
+ appenderator.close();
+ appenderatorClosedNormally = true;
+ }
+ finally {
+ if (!appenderatorClosedNormally) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator; continuing teardown.");
+ }
+ }
+ }
+
+ return TaskStatus.success(task.getId());
+ }
+
+ private boolean acquireLockForRestoredSegment(SegmentIdWithShardSpec segmentId)
+ {
+ try {
+ if (lockGranularity == LockGranularity.SEGMENT) {
+ return toolbox.getTaskActionClient().submit(
+ new SegmentLockAcquireAction(
+ lockType,
Review Comment:
[P2] Avoid APPEND locks with segment granularity
**Finding:** When forceTimeChunkLock=false, this task uses SEGMENT lock granularity but still defaults lockType to APPEND. The restored-segment path submits that unsupported pairing to SegmentLockAcquireAction, and SegmentAllocateAction receives the same pairing; TaskLockbox can reject it when an overlapping lock exists, preventing persisted segments from restoring or new batches from allocating under normal concurrent ingestion.
**Suggestion:** Use a supported lock-type/granularity pairing for segment mode and keep allocation, restoration, and publishing consistent, or explicitly reject forceTimeChunkLock=false; cover the configuration with a lockbox-backed test.
##########
docs/ingestion/kafka-share-group-ingestion.md:
##########
@@ -0,0 +1,349 @@
+---
+id: kafka-share-group-ingestion
+title: "Kafka share group ingestion"
+sidebar_label: "Kafka share group ingestion"
+description: "Queue-semantics ingestion from Apache Kafka using share groups (KIP-932). Scale consumers beyond partition count with at-least-once delivery."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+Requires Apache Kafka 4.0 or higher with share groups (KIP-932) enabled on the broker.
+:::
+
+## Overview
+
+Kafka share groups (KIP-932) let multiple consumers read from the same partition concurrently. The broker manages per-record acquisition locks and explicit acknowledgement, so consumer count is not capped by partition count, joining or leaving consumers does not pause the group, and a slow record does not block its partition.
+
+Druid's `ShareGroupIndexTask` consumes from a share group and publishes segments with at-least-once delivery: records are acknowledged only after their segments are atomically registered in the metadata store.
+
+## When to use share group ingestion
+
+| Scenario | Consumer group | Share group |
+|----------|---------------|-------------|
+| Workers needed exceed partition count | Idle workers | All workers active |
+| Elastic scaling (auto-scale events) | Rebalancing pause (30-60s) | Zero pause |
+| Per-message processing time varies | Head-of-line blocking | Independent processing |
+| Ordered processing required per partition | Yes | No (delivery order not guaranteed) |
+
+Choose share groups when throughput and elastic scaling matter more than strict per-partition ordering.
+
+## Task spec
+
+Submit a `ShareGroupIndexTask` to the Overlord. There are no start/end offsets -- the broker tracks them.
+
+```json
+{
+ "type": "index_kafka_share_group",
+ "dataSchema": {
+ "dataSource": "my_datasource",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "useSchemaDiscovery": true
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": "NONE"
+ }
+ },
+ "ioConfig": {
+ "topic": "my_topic",
+ "groupId": "druid-share-group",
+ "consumerProperties": {
+ "bootstrap.servers": "kafka-broker:9092"
+ },
+ "inputFormat": {
+ "type": "json"
+ },
+ "pollTimeout": 2000
+ },
+ "tuningConfig": {
+ "type": "KafkaTuningConfig",
+ "maxRowsPerSegment": 5000000
+ }
+}
+```
+
+## IO configuration
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `topic` | String | Yes | -- | Kafka topic to consume from. |
+| `groupId` | String | Yes | -- | Share group identifier. Multiple tasks with the same `groupId` share the workload. |
+| `consumerProperties` | Map | Yes | -- | Kafka consumer properties. Must include `bootstrap.servers`. See [Consumer property restrictions](#consumer-property-restrictions). |
+| `inputFormat` | Object | Yes | -- | Input format for parsing records (json, csv, avro, etc.). |
+| `pollTimeout` | Long | No | 2000 | Poll timeout in milliseconds. |
Review Comment:
[P3] Document the actual pollTimeout default
**Finding:** The configuration table says an omitted pollTimeout defaults to 2000 ms, but ShareGroupIndexTaskIOConfig delegates to KafkaSupervisorIOConfig.DEFAULT_POLL_TIMEOUT_MILLIS, which is 100 ms. Users following this documentation will poll twenty times more frequently than the documented default and may tune batch timing against the wrong value.
**Suggestion:** Make the documentation match the implementation, or change the new configuration default and its test to the intended 2000 ms value.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentLockAcquireAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.TimeChunkLockAcquireAction;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final LockGranularity lockGranularity;
+ private final TaskLockType lockType;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ final boolean useTimeChunkLocks = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ );
+ this.lockGranularity = useTimeChunkLocks ? LockGranularity.TIME_CHUNK : LockGranularity.SEGMENT;
+ this.lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(this::acquireLockForRestoredSegment);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
+ );
+
+ if (!addResult.isOk()) {
+ // Throw without acknowledging so the broker redelivers this batch.
+ throw new ISE(
+ "Could not allocate segment for row with timestamp[%s]",
+ row.getTimestamp()
+ );
+ }
+
+ totalRowsIngested++;
+ midBatchPersistNeeded |= addResult.isPersistRequired();
+
+ if (!pushThresholdLogged && addResult.isPushRequired(
+ tuningConfig.getPartitionsSpec().getMaxRowsPerSegment(),
+ tuningConfig.getPartitionsSpec()
+ .getMaxTotalRowsOr(DynamicPartitionsSpec.DEFAULT_MAX_TOTAL_ROWS)
+ )) {
+ pushThresholdLogged = true;
+ log.info(
+ "Segment row threshold reached during batch (current segment[%s], rows[%d]); "
+ + "will publish at end-of-batch boundary.",
+ addResult.getSegmentIdentifier(),
+ addResult.getNumRowsInSegment()
+ );
+ }
+ }
+ }
+
+ if (midBatchPersistNeeded) {
+ driver.persist(committerSupplier.get());
+ }
+
+ driver.persist(committerSupplier.get());
+
+ final TransactionalSegmentPublisher publisher = new TransactionalSegmentPublisher()
+ {
+ @Override
+ public SegmentPublishResult publishAnnotatedSegments(
+ @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,
+ Set<DataSegment> segmentsToPush,
+ @Nullable Object commitMetadata,
+ SegmentSchemaMapping segmentSchemaMapping
+ ) throws IOException
+ {
+ if (segmentsToPush.isEmpty()) {
+ return SegmentPublishResult.ok(segmentsToPush);
+ }
+ return toolbox.getTaskActionClient().submit(
+ SegmentTransactionalAppendAction.forSegments(segmentsToPush, segmentSchemaMapping)
Review Comment:
[P2] Select the publisher for the configured lock type
**Finding:** ShareGroupIndexTask accepts contexts with useConcurrentLocks=false or an explicit non-APPEND lock type, and the runner uses that type for allocation and restoration. This publisher always submits SegmentTransactionalAppendAction, whose perform method rejects every task lock that is not APPEND, so the first non-empty batch fails after persisting and its records remain unacknowledged for an otherwise accepted task configuration.
**Suggestion:** Reject or normalize non-APPEND lock contexts for this task, or select the publish action from the effective lock type as the existing batch-task path does; add coverage for useConcurrentLocks=false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4111030806
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @BeforeEach
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @AfterEach
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assertions.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled1);
+ Assertions.assertEquals("test-topic", polled1.getStream());
+ Assertions.assertEquals(0, polled1.getPartitionId().partition());
+ Assertions.assertNotNull(polled1.getData());
+ Assertions.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled2);
+ Assertions.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(1, result.size());
+ Assertions.assertTrue(result.get(0).getData().isEmpty());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assertions.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+ }
+
+ @Test
+ public void testCommitSync()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.empty());
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assertions.assertEquals(1, result.size());
+
+ final Map.Entry<KafkaTopicPartition, Optional<Exception>> entry = result.entrySet().iterator().next();
+ Assertions.assertEquals(Optional.of("test-topic"), entry.getKey().topic());
+ Assertions.assertEquals(0, entry.getKey().partition());
+ Assertions.assertFalse(entry.getValue().isPresent());
+ }
+
+ @Test
+ public void testCommitSyncWithError()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final KafkaException error = new KafkaException("commit failed");
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.of(error));
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assertions.assertEquals(1, result.size());
+
+ final Optional<Exception> maybeError = result.values().iterator().next();
+ Assertions.assertTrue(maybeError.isPresent());
+ Assertions.assertEquals("commit failed", maybeError.get().getMessage());
+ }
+
+ @Test
+ public void testCloseIsIdempotent()
+ {
+ supplier.close();
+ supplier.close();
+ verify(mockConsumer, Mockito.times(1)).close();
+ }
+
+ @Test
+ public void testWakeupForwardsToConsumer()
+ {
+ supplier.wakeup();
+ Mockito.verify(mockConsumer).wakeup();
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsForwardsToConsumer()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.of(45_000));
+ final Optional<Integer> lockMs = supplier.acquisitionLockTimeoutMs();
+ Assertions.assertTrue(lockMs.isPresent());
+ Assertions.assertEquals(Integer.valueOf(45_000), lockMs.get());
+ }
+
+ @Test
+ public void testAcknowledgeTwoArgOverloadDefaultsToAccept() throws Exception
+ {
+ final String topic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ topic,
+ 0,
+ 100L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(topic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, topic, 0), 100L);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testGetPartitionIdsReturnsEmptySet()
+ {
+ Assertions.assertEquals(Set.of(), supplier.getPartitionIds("any-stream"));
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsEmpty()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.empty());
+ Assertions.assertTrue(supplier.acquisitionLockTimeoutMs().isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeMap_multiPartition_acknowledgesAllRecords()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> r0 = new ConsumerRecord<>(testTopic, 0, 10L,
+ "k0".getBytes(StandardCharsets.UTF_8), "v0".getBytes(StandardCharsets.UTF_8));
+ final ConsumerRecord<byte[], byte[]> r1 = new ConsumerRecord<>(testTopic, 1, 20L,
+ "k1".getBytes(StandardCharsets.UTF_8), "v1".getBytes(StandardCharsets.UTF_8));
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(r0));
+ recordMap.put(new TopicPartition(testTopic, 1), List.of(r1));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000L);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(new KafkaTopicPartition(true, testTopic, 0), List.of(10L));
+ offsets.put(new KafkaTopicPartition(true, testTopic, 1), List.of(20L));
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ verify(mockConsumer).acknowledge(r0, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ verify(mockConsumer).acknowledge(r1, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testAcknowledgeMap_unknownOffset_throwsIllegalStateException()
+ {
+ final String testTopic = "test-topic";
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ supplier.poll(1000L);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(new KafkaTopicPartition(true, testTopic, 0), List.of(99L));
+ try {
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+ Assertions.fail("expected IllegalStateException for unknown offset");
+ }
+ catch (IllegalStateException expected) {
+ Assertions.assertTrue(expected.getMessage().contains("Cannot acknowledge unknown record"));
+ }
+ }
+
+ @Test
+ public void testAcknowledgeWithRenewType()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, testTopic, 0), 200L, AcknowledgeType.RENEW);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.RENEW);
+ }
+
+ @Test
+ public void testRenewedAppearanceReplacesAcknowledgementHandle()
+ {
+ final String testTopic = "test-topic";
+ final TopicPartition topicPartition = new TopicPartition(testTopic, 0);
+ final ConsumerRecord<byte[], byte[]> firstAppearance = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> renewedAppearance = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ when(mockConsumer.poll(any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(Map.of(topicPartition, List.of(firstAppearance))))
+ .thenReturn(new ConsumerRecords<>(Map.of(topicPartition, List.of(renewedAppearance))));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/12010)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @BeforeEach
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @AfterEach
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assertions.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled1);
+ Assertions.assertEquals("test-topic", polled1.getStream());
+ Assertions.assertEquals(0, polled1.getPartitionId().partition());
+ Assertions.assertNotNull(polled1.getData());
+ Assertions.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled2);
+ Assertions.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(1, result.size());
+ Assertions.assertTrue(result.get(0).getData().isEmpty());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assertions.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+ }
+
+ @Test
+ public void testCommitSync()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.empty());
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assertions.assertEquals(1, result.size());
+
+ final Map.Entry<KafkaTopicPartition, Optional<Exception>> entry = result.entrySet().iterator().next();
+ Assertions.assertEquals(Optional.of("test-topic"), entry.getKey().topic());
+ Assertions.assertEquals(0, entry.getKey().partition());
+ Assertions.assertFalse(entry.getValue().isPresent());
+ }
+
+ @Test
+ public void testCommitSyncWithError()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final KafkaException error = new KafkaException("commit failed");
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.of(error));
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assertions.assertEquals(1, result.size());
+
+ final Optional<Exception> maybeError = result.values().iterator().next();
+ Assertions.assertTrue(maybeError.isPresent());
+ Assertions.assertEquals("commit failed", maybeError.get().getMessage());
+ }
+
+ @Test
+ public void testCloseIsIdempotent()
+ {
+ supplier.close();
+ supplier.close();
+ verify(mockConsumer, Mockito.times(1)).close();
+ }
+
+ @Test
+ public void testWakeupForwardsToConsumer()
+ {
+ supplier.wakeup();
+ Mockito.verify(mockConsumer).wakeup();
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsForwardsToConsumer()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.of(45_000));
+ final Optional<Integer> lockMs = supplier.acquisitionLockTimeoutMs();
+ Assertions.assertTrue(lockMs.isPresent());
+ Assertions.assertEquals(Integer.valueOf(45_000), lockMs.get());
+ }
+
+ @Test
+ public void testAcknowledgeTwoArgOverloadDefaultsToAccept() throws Exception
+ {
+ final String topic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ topic,
+ 0,
+ 100L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(topic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, topic, 0), 100L);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testGetPartitionIdsReturnsEmptySet()
+ {
+ Assertions.assertEquals(Set.of(), supplier.getPartitionIds("any-stream"));
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsEmpty()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.empty());
+ Assertions.assertTrue(supplier.acquisitionLockTimeoutMs().isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeMap_multiPartition_acknowledgesAllRecords()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> r0 = new ConsumerRecord<>(testTopic, 0, 10L,
+ "k0".getBytes(StandardCharsets.UTF_8), "v0".getBytes(StandardCharsets.UTF_8));
+ final ConsumerRecord<byte[], byte[]> r1 = new ConsumerRecord<>(testTopic, 1, 20L,
+ "k1".getBytes(StandardCharsets.UTF_8), "v1".getBytes(StandardCharsets.UTF_8));
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(r0));
+ recordMap.put(new TopicPartition(testTopic, 1), List.of(r1));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000L);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(new KafkaTopicPartition(true, testTopic, 0), List.of(10L));
+ offsets.put(new KafkaTopicPartition(true, testTopic, 1), List.of(20L));
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ verify(mockConsumer).acknowledge(r0, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ verify(mockConsumer).acknowledge(r1, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testAcknowledgeMap_unknownOffset_throwsIllegalStateException()
+ {
+ final String testTopic = "test-topic";
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ supplier.poll(1000L);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(new KafkaTopicPartition(true, testTopic, 0), List.of(99L));
+ try {
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+ Assertions.fail("expected IllegalStateException for unknown offset");
+ }
+ catch (IllegalStateException expected) {
+ Assertions.assertTrue(expected.getMessage().contains("Cannot acknowledge unknown record"));
+ }
+ }
+
+ @Test
+ public void testAcknowledgeWithRenewType()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, testTopic, 0), 200L, AcknowledgeType.RENEW);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.RENEW);
+ }
+
+ @Test
+ public void testRenewedAppearanceReplacesAcknowledgementHandle()
+ {
+ final String testTopic = "test-topic";
+ final TopicPartition topicPartition = new TopicPartition(testTopic, 0);
+ final ConsumerRecord<byte[], byte[]> firstAppearance = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> renewedAppearance = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ when(mockConsumer.poll(any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(Map.of(topicPartition, List.of(firstAppearance))))
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/12011)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3195460477
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Sure, I will keep in mind. @FrankChen021 after these changes should we go ahead and phase wise rollout the kafka share group implementation in Druid ?
I will address the review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3195642219
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Phasing the implementation sounds reasonable to me, especially if supervisor and dedup-cache work makes the PR too large to review as one change. But I would still treat this specific issue as a merge blocker unless this phase either fixes it or explicitly scopes it out in code/docs: the current runner still ACKs the whole Kafka record after publish while `parseRow()` only consumes the first row from the input reader. That means the no-data-loss invariant is still inaccurate for any input format that can produce multiple rows from one Kafka record. For this phase, I think the minimum viable resolution is either to iterate and add every row before ACKing, or to reject/document a one-row-per-record limitation so users cannot assume the stronger invariant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3273016198
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexerCrashIT.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Verifies at-least-once semantics: uncommitted records are redelivered after Indexer crash.
+ */
+public class ShareGroupIndexerCrashIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+ private static final String GROUP_ID = "crash-it-group";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(60)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_indexerCrash_midIngestion_noDataLossAfterRecovery() throws Exception
+ {
+ final String topic = dataSource + "_crash_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset(GROUP_ID, "earliest");
+
+ // Produce all records before starting task so Indexer is polling them when it crashes.
+ final int batchA = 20;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(batchA, 0, "2025-09-01"));
+
+ // Task 1: starts ingesting but Indexer crashes before it completes.
+ final String taskId1 = submitTask(topic);
Review Comment:
## CodeQL / Unread local variable
Variable 'String taskId1' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/11242)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3298287473
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunnerTest.java:
##########
@@ -271,6 +272,61 @@ public void testRunLoop_segmentAllocationFails_throwsAndExits() throws Exception
}
}
+ @Test
+ public void testRunLoop_concurrentTasks_useDifferentSequenceNames() throws Exception
+ {
+ final ShareGroupIndexTask taskA = buildTask("share_group_task_A");
+ final ShareGroupIndexTask taskB = buildTask("share_group_task_B");
+ final java.util.Set<String> seqNamesUsedByA = runOnceAndCaptureSequenceNames(taskA);
+ final java.util.Set<String> seqNamesUsedByB = runOnceAndCaptureSequenceNames(taskB);
+ Assert.assertFalse("Task A should call driver.add at least once", seqNamesUsedByA.isEmpty());
+ Assert.assertFalse("Task B should call driver.add at least once", seqNamesUsedByB.isEmpty());
+ final java.util.Set<String> shared = new java.util.HashSet<>(seqNamesUsedByA);
+ shared.retainAll(seqNamesUsedByB);
+ Assert.assertTrue(
+ "Concurrent share-group tasks must not share segment sequence names; shared=" + shared,
Review Comment:
validating that seq name is different for both task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3259806882
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Throwables;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/** Verifies that ShareGroupIndexTask can be submitted via raw JSON through the Overlord HTTP API. */
+public class ShareGroupIndexTaskJsonSubmitIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_jsonSubmit_csvFormat_ingestsData() throws Exception
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+ kafkaServer.setShareGroupAutoOffsetReset("json-submit-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "json-submit-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 8;
+ kafkaServer.publishRecordsToTopic(topic, generateRecords(numRecords));
+
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(numRecords)
+ );
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ final long deadlineMs = System.currentTimeMillis() + 30_000L;
+ while (System.currentTimeMillis() < deadlineMs) {
+ try {
+ cluster.callApi().verifySqlQuery("SELECT COUNT(*) FROM %s", dataSource, String.valueOf(numRecords));
+ break;
+ }
+ catch (AssertionError ignored) {
+ Thread.sleep(500);
+ }
+ }
+
+ cluster.callApi().onLeaderOverlord(o -> o.cancelTask(task.getId()));
+ cluster.callApi().waitForTaskToFinish(task.getId(), overlord.latchableEmitter());
+ }
+
+ @Test
+ public void test_jsonSubmit_roundTripsPayload() throws Exception
+ {
+ final String topic = dataSource + "_roundtrip_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset("roundtrip-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "roundtrip-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final org.apache.druid.client.indexing.TaskPayloadResponse payloadResponse =
+ cluster.callApi().onLeaderOverlord(o -> o.taskPayload(task.getId()));
+ Assertions.assertEquals(task.getId(), payloadResponse.getTask());
+
+ final JsonNode payloadJson = mapper.valueToTree(payloadResponse.getPayload());
+ Assertions.assertEquals("share_group_index", payloadJson.path("type").asText());
Review Comment:
https://github.com/apache/druid/pull/19311/changes/872d6f3b93031491ed35811765e0291bcbf98bd6#diff-c24dbac31b9f7e4062cd05cd6260d05c23c1c49b1c15ab8df3f9813efd806cb6
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Throwables;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/** Verifies that ShareGroupIndexTask can be submitted via raw JSON through the Overlord HTTP API. */
+public class ShareGroupIndexTaskJsonSubmitIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_jsonSubmit_csvFormat_ingestsData() throws Exception
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+ kafkaServer.setShareGroupAutoOffsetReset("json-submit-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "json-submit-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 8;
+ kafkaServer.publishRecordsToTopic(topic, generateRecords(numRecords));
+
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(numRecords)
+ );
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ final long deadlineMs = System.currentTimeMillis() + 30_000L;
Review Comment:
https://github.com/apache/druid/pull/19311/changes/872d6f3b93031491ed35811765e0291bcbf98bd6#diff-c24dbac31b9f7e4062cd05cd6260d05c23c1c49b1c15ab8df3f9813efd806cb6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4729631553
Hi @FrankChen021 , We have added IT , unit tests : embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedShareGroupIngestionTest.java - please let me know if we can get this PR in and Druid will be among first who have share group implementation available and then focus on enhancements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4032533135
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(segmentId -> true);
Review Comment:
Updated https://github.com/apache/druid/pull/19311/commits/8341dc6f352eea13572598e196800c425f8d94b8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4133565073
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/DurableShareGroupBatchStager.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.common.actions.StageShareInboxBatchAction;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.overlord.ShareInboxBatch;
+import org.apache.druid.indexing.overlord.ShareInboxStageResult;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+
+final class DurableShareGroupBatchStager implements ShareGroupBatchStager
+{
+ private static final Logger log = new Logger(DurableShareGroupBatchStager.class);
+
+ private final ShareGroupBatchStore batchStore;
+ private final TaskActionClient taskActionClient;
+ private final String dataSource;
+ private final String inboxId;
+ private final String groupId;
+ private final ShareGroupSourceIdentity sourceIdentity;
+ private final String specFingerprint;
+ private final int receiptPageSize;
+
+ DurableShareGroupBatchStager(
+ ShareGroupBatchStore batchStore,
+ TaskActionClient taskActionClient,
+ String dataSource,
+ String inboxId,
+ String groupId,
+ ShareGroupSourceIdentity sourceIdentity,
+ String specFingerprint,
+ int receiptPageSize
+ )
+ {
+ this.batchStore = Preconditions.checkNotNull(batchStore, "batchStore");
+ this.taskActionClient = Preconditions.checkNotNull(taskActionClient, "taskActionClient");
+ this.dataSource = Preconditions.checkNotNull(dataSource, "dataSource");
+ this.inboxId = Preconditions.checkNotNull(inboxId, "inboxId");
+ this.groupId = Preconditions.checkNotNull(groupId, "groupId");
+ this.sourceIdentity = Preconditions.checkNotNull(sourceIdentity, "sourceIdentity");
+ this.specFingerprint = Preconditions.checkNotNull(specFingerprint, "specFingerprint");
+ Preconditions.checkArgument(receiptPageSize > 0, "receiptPageSize must be positive");
+ this.receiptPageSize = receiptPageSize;
+ }
+
+ @Override
+ public void stage(
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records
+ ) throws IOException
+ {
+ final ShareGroupBatchStore.StoredBatch storedBatch = batchStore.store(records);
+ final ShareGroupRawBatch rawBatch = storedBatch.getBatch();
+ final List<Long> offsets = rawBatch.getRecords().stream()
+ .map(ShareGroupRawRecord::getOffset)
+ .toList();
+ final ShareInboxBatch inboxBatch = new ShareInboxBatch(
+ dataSource,
+ inboxId,
+ groupId,
+ sourceIdentity.getClusterId(),
+ sourceIdentity.getTopicId(),
+ sourceIdentity.getTopicName(),
+ rawBatch.getPartition(),
+ specFingerprint,
+ UUID.randomUUID().toString(),
+ storedBatch.getPath(),
+ storedBatch.getSha256(),
+ storedBatch.getSize(),
+ offsets,
+ receiptPageSize
+ );
+ final ShareInboxStageResult result = taskActionClient.submit(new StageShareInboxBatchAction(inboxBatch));
+ validateResult(result, offsets);
+ if (result.getStatus() == ShareInboxStageResult.Status.ALREADY_STAGED) {
+ try {
+ batchStore.delete(storedBatch);
Review Comment:
[P1] Preserve the referenced object when a stage request is retried
**Finding:** If the Overlord commits StageShareInboxBatchAction but its response is lost, RemoteTaskActionClient can automatically retry the same request. stageShareInboxBatch then finds the receipts and returns ALREADY_STAGED even though the existing manifest references this exact storedBatch path (the coordinator test explicitly expects this result for a repeated batch). This branch deletes that referenced object and staging reports success, allowing ACCEPT to be sent to Kafka. If inbox processing has not read the object yet, the acknowledged records can no longer be recovered. The lost-response unit test only models an exception escaping submit, so it misses the successful internal retry.
**Suggestion:** Make staging retries idempotent by manifest ID and distinguish an already referenced object from a redundant new upload before deleting anything; cover commit followed by a lost response and successful retry.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/AppenderatorShareInboxBatchHandler.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Supplier;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendFromShareInboxAction;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.overlord.ShareInboxCompletionRequest;
+import org.apache.druid.indexing.overlord.ShareInboxManifest;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.UUID;
+
+final class AppenderatorShareInboxBatchHandler implements ShareInboxBatchHandler
+{
+ private final StreamAppenderatorDriver driver;
+ private final StreamChunkReader<KafkaRecordEntity> chunkReader;
+ private final TaskActionClient taskActionClient;
+ private final String taskId;
+
+ AppenderatorShareInboxBatchHandler(
+ StreamAppenderatorDriver driver,
+ StreamChunkReader<KafkaRecordEntity> chunkReader,
+ TaskActionClient taskActionClient,
+ String taskId
+ )
+ {
+ this.driver = driver;
+ this.chunkReader = chunkReader;
+ this.taskActionClient = taskActionClient;
+ this.taskId = taskId;
+ }
+
+ @Override
+ public void process(
+ List<ShareInboxManifest> manifests,
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records,
+ ShareInboxClaimLease claimLease
+ ) throws Exception
+ {
+ final String completionId = UUID.randomUUID().toString();
+ final String sequenceName = taskId + "-share-inbox-" + completionId;
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return Map.of("shareInboxCompletionId", completionId);
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ boolean rowsAdded = false;
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ claimLease.assertValid();
+ final List<InputRow> rows = chunkReader.parse(kafkaEntities(record), false);
+ for (InputRow row : rows) {
+ claimLease.assertValid();
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
Review Comment:
[P2] Honor the appenderator persistence threshold inside the row loop
**Finding:** driver.add is called with allowIncrementalPersists=false, but the returned isPersistRequired signal is never handled. The only persist happens after every record in the claim has been parsed and added. A valid claim can contain up to 100,000 Kafka records by default, and each record can produce multiple rows, so maxRowsInMemory/maxBytesInMemory can be exceeded for the entire batch. Large claims or a deliberately small memory setting can exhaust the task heap and fail repeatedly on the same durable inbox batch.
**Suggestion:** Enable incremental persists or persist as soon as addResult.isPersistRequired() becomes true, and verify a multi-row batch that crosses the configured memory threshold.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/AppenderatorShareInboxBatchHandler.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Supplier;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendFromShareInboxAction;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.overlord.ShareInboxCompletionRequest;
+import org.apache.druid.indexing.overlord.ShareInboxManifest;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.UUID;
+
+final class AppenderatorShareInboxBatchHandler implements ShareInboxBatchHandler
+{
+ private final StreamAppenderatorDriver driver;
+ private final StreamChunkReader<KafkaRecordEntity> chunkReader;
+ private final TaskActionClient taskActionClient;
+ private final String taskId;
+
+ AppenderatorShareInboxBatchHandler(
+ StreamAppenderatorDriver driver,
+ StreamChunkReader<KafkaRecordEntity> chunkReader,
+ TaskActionClient taskActionClient,
+ String taskId
+ )
+ {
+ this.driver = driver;
+ this.chunkReader = chunkReader;
+ this.taskActionClient = taskActionClient;
+ this.taskId = taskId;
+ }
+
+ @Override
+ public void process(
+ List<ShareInboxManifest> manifests,
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records,
+ ShareInboxClaimLease claimLease
+ ) throws Exception
+ {
+ final String completionId = UUID.randomUUID().toString();
+ final String sequenceName = taskId + "-share-inbox-" + completionId;
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return Map.of("shareInboxCompletionId", completionId);
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ boolean rowsAdded = false;
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ claimLease.assertValid();
+ final List<InputRow> rows = chunkReader.parse(kafkaEntities(record), false);
+ for (InputRow row : rows) {
+ claimLease.assertValid();
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ sequenceName,
+ committerSupplier,
+ true,
+ false
+ );
+ if (!addResult.isOk()) {
+ throw new ISE("Could not allocate segment for share inbox row with timestamp[%s]", row.getTimestamp());
+ }
+ rowsAdded = true;
+ }
+ }
+ if (rowsAdded) {
+ claimLease.assertValid();
+ driver.persist(committerSupplier.get());
+ }
+
+ final ShareInboxCompletionRequest completionRequest = completionRequest(manifests, completionId);
+ final TransactionalSegmentPublisher publisher = new TransactionalSegmentPublisher()
+ {
+ @Override
+ public SegmentPublishResult publishAnnotatedSegments(
+ @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,
+ Set<DataSegment> segmentsToPush,
+ @Nullable Object commitMetadata,
+ SegmentSchemaMapping segmentSchemaMapping
+ ) throws IOException
+ {
+ claimLease.assertValid();
+ return taskActionClient.submit(
+ new SegmentTransactionalAppendFromShareInboxAction(
+ segmentsToPush,
+ completionRequest,
+ segmentSchemaMapping
+ )
+ );
+ }
+
+ @Override
+ public boolean supportsEmptyPublish()
+ {
+ return true;
+ }
+ };
+
+ claimLease.assertValid();
+ driver.publish(publisher, committerSupplier.get(), List.of(sequenceName)).get();
Review Comment:
[P2] Hand published segments off before retaining the next batches
**Finding:** The publish future is awaited and discarded, and no share-group path calls registerHandoff or drops the published segments. StreamAppenderatorDriver.publish removes sequence tracking, but it does not remove the underlying StreamAppenderator sinks; registerHandoff is the path that drops them once a Historical serves them. Since this task loops indefinitely and creates a fresh sequence for each claim, completed batches retain their persisted files, mapped indexes, and sink bookkeeping until task termination. A normally running ingestion task therefore grows local resource usage without bound even after the segments are available on Historicals.
**Suggestion:** Register handoff using the publication result and manage the resulting futures so completed segments are dropped and handoff failures are observed.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentTransactionalAppendFromShareInboxAction.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.PendingSegmentAllocatingTask;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.CriticalAction;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.overlord.ShareInboxCompletionRequest;
+import org.apache.druid.metadata.ReplaceTaskLock;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.timeline.DataSegment;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class SegmentTransactionalAppendFromShareInboxAction implements TaskAction<SegmentPublishResult>
+{
+ public static final String TYPE = "segmentTransactionalAppendFromShareInbox";
+
+ private final Set<DataSegment> segments;
+ private final ShareInboxCompletionRequest completionRequest;
+ @Nullable
+ private final SegmentSchemaMapping segmentSchemaMapping;
+
+ @JsonCreator
+ public SegmentTransactionalAppendFromShareInboxAction(
+ @JsonProperty("segments") Set<DataSegment> segments,
+ @JsonProperty("completionRequest") ShareInboxCompletionRequest completionRequest,
+ @JsonProperty("segmentSchemaMapping") @Nullable SegmentSchemaMapping segmentSchemaMapping
+ )
+ {
+ this.segments = Set.copyOf(Objects.requireNonNull(segments, "segments"));
+ this.completionRequest = Objects.requireNonNull(completionRequest, "completionRequest");
+ this.segmentSchemaMapping = segmentSchemaMapping;
+ }
+
+ @JsonProperty
+ public Set<DataSegment> getSegments()
+ {
+ return segments;
+ }
+
+ @JsonProperty
+ public ShareInboxCompletionRequest getCompletionRequest()
+ {
+ return completionRequest;
+ }
+
+ @Nullable
+ @JsonProperty
+ public SegmentSchemaMapping getSegmentSchemaMapping()
+ {
+ return segmentSchemaMapping;
+ }
+
+ @Override
+ public TypeReference<SegmentPublishResult> getReturnTypeReference()
+ {
+ return new TypeReference<>() {};
+ }
+
+ @Override
+ public SegmentPublishResult perform(Task task, TaskActionToolbox toolbox)
+ {
+ if (!(task instanceof PendingSegmentAllocatingTask)) {
+ throw DruidException.defensive(
+ "Task[%s] of type[%s] cannot append share inbox segments because it does not implement PendingSegmentAllocatingTask.",
+ task.getId(),
+ task.getType()
+ );
+ }
+ ClaimShareInboxManifestsAction.validateTask(
+ task,
+ completionRequest.getDataSource(),
+ completionRequest.getTaskId()
+ );
+
+ final List<TaskLock> locks = toolbox.getTaskLockbox().findLocksForTask(task);
+ for (TaskLock lock : locks) {
+ if (lock.getType() != TaskLockType.APPEND) {
Review Comment:
[P2] Keep the accepted task lock context compatible with publication
**Finding:** The prior non-APPEND lock finding remains in the new inbox publisher. ShareGroupIndexTask preserves an explicit useConcurrentLocks=false context, and the runner then allocates with EXCLUSIVE or SHARED locks through determineLockTypeForAppend. This action rejects those locks unconditionally. The accepted task consequently stages and acknowledges Kafka records but fails at its first non-empty segment publication; retrying the same configuration cannot drain the inbox.
**Suggestion:** Reject or normalize non-APPEND contexts before acquisition starts, or implement a completion publisher compatible with the effective lock type and test useConcurrentLocks=false.
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -163,6 +179,590 @@ public Set<String> retrieveAllDatasourceNames()
);
}
+ @Override
+ public ShareInboxStageResult stageShareInboxBatch(ShareInboxBatch batch)
+ {
+ return connector.retryTransaction(
+ (handle, status) -> {
+ try {
+ return stageShareInboxBatch(handle, batch);
+ }
+ catch (RuntimeException e) {
+ if (connector.isUniqueConstraintViolation(e)) {
+ throw new RetryTransactionException("Concurrent share inbox receipt update");
+ }
+ throw e;
+ }
+ },
+ SQLMetadataConnector.QUIET_RETRIES,
+ SQLMetadataConnector.DEFAULT_MAX_TRIES
+ );
+ }
+
+ @Override
+ public ShareInboxClaimResult claimShareInboxManifests(ShareInboxClaimRequest request)
+ {
+ return connector.retryTransaction(
+ (handle, status) -> claimShareInboxManifests(handle, request),
+ SQLMetadataConnector.QUIET_RETRIES,
+ SQLMetadataConnector.DEFAULT_MAX_TRIES
+ );
+ }
+
+ private ShareInboxClaimResult claimShareInboxManifests(Handle handle, ShareInboxClaimRequest request)
+ {
+ validateShareInboxGeneration(
+ handle,
+ request.getDataSource(),
+ request.getInboxId(),
+ request.getSpecFingerprint()
+ );
+ final DateTime now = DateTimes.nowUtc();
+ final String nowString = now.toString();
+ final String claimExpiresAt = now.plus(request.getClaimDurationMillis()).toString();
+ final List<ShareInboxCandidate> candidates = handle.createQuery(
+ StringUtils.format(
+ "SELECT manifest_id, object_size, record_count, claim_epoch FROM %s "
+ + "WHERE data_source = :dataSource AND inbox_id = :inboxId "
+ + "AND spec_fingerprint = :specFingerprint "
+ + "AND (state = 'STAGED' OR (state = 'CLAIMED' AND claim_expires_at <= :now)) "
+ + "ORDER BY created_date, manifest_id %s",
+ dbTables.getShareInboxTable(),
+ connector.limitClause(request.getMaxManifests())
+ )
+ )
+ .bind("dataSource", request.getDataSource())
+ .bind("inboxId", request.getInboxId())
+ .bind("specFingerprint", request.getSpecFingerprint())
+ .bind("now", nowString)
+ .map(
+ (index, resultSet, context) ->
+ new ShareInboxCandidate(
+ resultSet.getString("manifest_id"),
+ resultSet.getLong("object_size"),
+ resultSet.getInt("record_count"),
+ resultSet.getLong("claim_epoch")
+ )
+ )
+ .list();
+
+ final List<ShareInboxManifest> claimed = new ArrayList<>();
+ long claimedBytes = 0;
+ int claimedRecords = 0;
+ for (ShareInboxCandidate candidate : candidates) {
+ if (claimedBytes + candidate.objectSize > request.getMaxBytes()
Review Comment:
[P2] Ensure every staged manifest can fit the processing limits
**Finding:** A manifest larger than maxRecords or maxBytes is skipped even when the claim is empty. Staging uses separate maxStagingRecords/maxStagingBytes limits and never splits a partition batch to the processing limits. For example, maxProcessingRecords=100 with a 500-record polled partition batch stages and acknowledges all 500 records, but that manifest is never claimable. Once the oldest maxManifests candidates are oversized, the SQL LIMIT also prevents later smaller manifests from being considered, so the inbox can stop progressing indefinitely.
**Suggestion:** Split or validate staged batches against processing bounds, or allow one oversized manifest to make progress, and ensure oversized candidates cannot permanently hide later work.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,618 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.SegmentLockAcquireAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.TimeChunkLockAcquireAction;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.storage.StorageConnector;
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Runs durable acquisition and inbox processing for {@link ShareGroupIndexTask}.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final long DEFAULT_MAX_POLL_INTERVAL_MS = 300_000L;
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final LockGranularity lockGranularity;
+ private final TaskLockType lockType;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+ private final Function<ShareGroupIndexTaskIOConfig, ShareGroupSourceIdentity> sourceIdentityFactory;
+
+ @Nullable
+ private ShareGroupSourceIdentity sourceIdentity;
+ @Nullable
+ private String specFingerprint;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+ private final AtomicReference<ShareGroupAcquisitionLoop> activeAcquisitionLoop = new AtomicReference<>();
+ private final AtomicReference<ShareInboxProcessor> activeInboxProcessor = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this(task, toolbox, configMapper, supplierFactory, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory,
+ @Nullable Function<ShareGroupIndexTaskIOConfig, ShareGroupSourceIdentity> sourceIdentityFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ final boolean useTimeChunkLocks = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ );
+ this.lockGranularity = useTimeChunkLocks ? LockGranularity.TIME_CHUNK : LockGranularity.SEGMENT;
Review Comment:
[P2] Reject the unsupported APPEND and segment-lock combination
**Finding:** The prior lock-granularity finding is still present: forceTimeChunkLock=false selects SEGMENT while the default concurrent-lock context selects APPEND. Both allocation and restored-segment acquisition receive that pair. TaskLockbox explicitly rejects APPEND/REPLACE requests with non-TIME_CHUNK granularity when an overlapping non-reusable lock exists, so running this accepted configuration alongside ordinary concurrent ingestion or replacement can prevent allocation or restoration. The restored-lock unit tests mock the action result and therefore do not exercise this lockbox restriction.
**Suggestion:** Require time-chunk APPEND locks for this task or implement a supported alternative consistently across allocation, restoration, and inbox publication, with a lockbox-backed test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3182298828
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Thanks for checking. Yes, in phase 2 implementation I am trying to make sure no data loss.
Also trying couple of benchmark when Kafka Queue Semantics actually benefit us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3259684199
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexerCrashIT.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Verifies at-least-once semantics: uncommitted records are redelivered after Indexer crash.
+ */
Review Comment:
Validation of no data loss in crash situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3280204201
##########
indexing-service/src/test/java/org/apache/druid/indexing/seekablestream/common/AcknowledgeTypeTest.java:
##########
@@ -42,4 +49,69 @@ public void testValueOfRoundTripsForEachConstant()
Assert.assertSame(type, AcknowledgeType.valueOf(type.name()));
}
}
+
+ @Test
+ public void testAcknowledgingRecordSupplierDefaultMethods()
+ {
+ final AcknowledgingRecordSupplier<String, Long, ByteEntity> stub =
+ new AcknowledgingRecordSupplier<String, Long, ByteEntity>()
+ {
+ @Override
+ public void subscribe(Set<String> topics)
+ {
+ }
+
+ @Override
+ public void unsubscribe()
+ {
+ }
+
+ @Override
+ public Set<String> subscription()
+ {
+ return Set.of();
+ }
+
+ @Override
+ public List<OrderedPartitionableRecord<String, Long, ByteEntity>> poll(long timeoutMs)
+ {
+ return List.of();
+ }
+
+ @Override
+ public void acknowledge(String partitionId, Long offset)
+ {
+ }
+
+ @Override
+ public void acknowledge(String partitionId, Long offset, AcknowledgeType type)
+ {
+ }
+
+ @Override
+ public void acknowledge(Map<String, Collection<Long>> offsets, AcknowledgeType type)
+ {
+ }
+
+ @Override
+ public Map<String, Optional<Exception>> commitSync()
+ {
+ return Map.of();
+ }
+
+ @Override
+ public Set<String> getPartitionIds(String stream)
+ {
+ return Set.of();
+ }
+
+ @Override
+ public void close()
+ {
+ }
+ };
+
+ stub.wakeup();
+ Assert.assertFalse(stub.acquisitionLockTimeoutMs().isPresent());
Review Comment:
Just to have test coverage CI check pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3298332942
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -121,6 +121,7 @@ public class ShareGroupIndexTaskRunner
this.task = task;
this.toolbox = toolbox;
this.configMapper = configMapper;
+ this.sequenceName = task.getId();
Review Comment:
task ID is unique so using it for sequence name @FrankChen021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3242933267
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assert.assertNotNull(polled2);
+ Assert.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11228)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11227)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3213018385
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTask.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.common.config.TaskConfig;
+import org.apache.druid.indexing.common.task.AbstractTask;
+import org.apache.druid.indexing.common.task.PendingSegmentAllocatingTask;
+import org.apache.druid.indexing.common.task.TaskResource;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.ResourceAction;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Indexing task that consumes from a Kafka topic with share-group semantics
+ * (KIP-932). The broker tracks delivery state; the task acknowledges records
+ * after the segments containing them are published. Phase 1: single-threaded,
+ * no supervisor, no dedup cache.
+ */
+public class ShareGroupIndexTask extends AbstractTask implements PendingSegmentAllocatingTask
Review Comment:
Updated https://github.com/apache/druid/pull/19311/commits/ede507254b92a60cef4fd174d10a5caf64dcca92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4026096327
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
Review Comment:
P1 Use JUnit 5 APIs for the new unit tests
**Finding:** This new test source imports org.junit.* (including the import at this line), and the same JUnit 4 API is used by ShareGroupConsumerPropertiesTest, ShareGroupIndexTaskIOConfigTest, ShareGroupIndexTaskRunnerTest, ShareGroupIndexTaskTest, and indexing-service's AcknowledgeTypeTest. The kafka-indexing-service and indexing-service POMs provide only JUnit Jupiter dependencies and no JUnit 4 or Vintage artifact, so normal test compilation cannot resolve these imports and the added tests cannot run.
**Suggestion:** Migrate all six new unit-test classes to the repository's JUnit 5 APIs, lifecycle annotations, and assertions, or add an explicitly supported JUnit 4/Vintage dependency if that compatibility is intentional.
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final String sequenceName;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.sequenceName = task.getId();
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(segmentId -> true);
Review Comment:
P2 Reacquire locks for restored appenderator segments
**Finding:** On a restart or retry with persisted appenderator state, StreamAppenderatorDriver.startJob invokes the supplied lock helper for every restored non-dropped segment. This unconditional true reports each segment as locked without acquiring the task's current append/time-chunk lock, so the task can resume or publish rows under a lock it does not own; when the previous lock is gone, the restored segment can be rejected or stranded instead of being safely resumed.
**Suggestion:** Pass a lock helper that acquires and verifies the same lock granularity and type used by SegmentAllocateAction, matching the existing SeekableStreamIndexTaskRunner startup path, and fail restoration when acquisition fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3195460477
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Sure, I will keep in mind. @FrankChen021 after these changes should we go ahead and phase wise rollout the kafka share group implementation in Druid ?
I will address this review comment and also the previous one
> Create a new NonSharedQueueKafkaIndexTask, which is exactly current implementation
create a new SharedQueueKafkaIndexTask, which is the one your want to implement
the KafkaIndexTask holds either instance of NonSharedQueueKafkaIndexTask or SharedQueueKafkaIndexTask based on the "type": "kafka_share_group" defined in the ioConfig, and delegate all public methods to these instance
I find that after implementing the supervisor and dedup cache (in my local branch) it has become a huge code changes that would be hard to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4404406499
For smaller blast radius and clean separation - parking the suggestion https://github.com/apache/druid/issues/18439#issuecomment-4294451459 , Also KafkaShareGroupSupervisor (planned) will be structurally different from KafkaSupervisor — no partition-to-task assignment, no offset bookkeeping, seek/rest .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3207054221
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedShareGroupIngestionTest.java:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.JsonInputFormat;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+/** End-to-end IT for Kafka share-group ingestion; requires Docker. */
+public class EmbeddedShareGroupIngestionTest extends EmbeddedClusterTestBase
+{
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private static final InputFormat DEFAULT_CSV_FORMAT =
+ new CsvInputFormat(List.of(COL_TIMESTAMP, COL_ITEM, COL_VALUE), null, null, false, 0, false);
+
+ /** Lets a freshly submitted share-group task reach STABLE before records are produced. */
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+
+ return cluster;
+ }
+
+ @Test
+ public void test_shareGroupIngestion_basicEndToEnd() throws InterruptedException
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+
+ final String taskId = submitShareGroupTask(
+ topic,
+ "druid-share-group-test",
+ kafkaServer.consumerProperties(),
+ DEFAULT_CSV_FORMAT,
+ defaultTuningConfig()
+ );
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 10;
+ kafkaServer.produceRecordsToTopic(
+ generateRecords(topic, numRecords, DateTimes.of("2025-06-01"))
+ );
+
+ waitForRowsProcessed(numRecords);
+
+ Assertions.assertEquals(
+ String.valueOf(numRecords),
+ cluster.runSql("SELECT COUNT(*) FROM %s", dataSource)
+ );
+
+ cancelAndAwaitTermination(taskId);
+ }
+
+ /** Verifies that one Kafka record carrying multiple JSON objects yields one row per object. */
+ @Test
+ public void test_shareGroupIngestion_multiRowPerKafkaRecord() throws InterruptedException
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+
+ final String taskId = submitShareGroupTask(
+ topic,
+ "druid-share-group-multirow-test",
+ kafkaServer.consumerProperties(),
+ new JsonInputFormat(null, Collections.emptyMap(), null, null, true),
+ defaultTuningConfig()
+ );
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 5;
+ final int rowsPerRecord = 4;
+ kafkaServer.produceRecordsToTopic(
+ generateMultiObjectJsonRecords(topic, numRecords, rowsPerRecord, DateTimes.of("2025-07-01"))
Review Comment:
StreamChunkReader multi-row path One Kafka record carrying 4 whitespace-separated JSON objects produces 4 rows. SQL count = numRecords × rowsPerRecord (5 × 4 = 20). Verifies no row is dropped before ACK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on PR #19311:
URL: https://github.com/apache/druid/pull/19311#issuecomment-4472179890
fixing the CI check for coverage improvements. Looks like it is bit tricky to unit test`extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java` run method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3280352075
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assert.assertNotNull(polled2);
+ Assert.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(1, result.size());
+ Assert.assertTrue(result.get(0).getData().isEmpty());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assert.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+ }
+
+ @Test
+ public void testCommitSync()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.empty());
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Map.Entry<KafkaTopicPartition, Optional<Exception>> entry = result.entrySet().iterator().next();
+ Assert.assertEquals(Optional.of("test-topic"), entry.getKey().topic());
+ Assert.assertEquals(0, entry.getKey().partition());
+ Assert.assertFalse(entry.getValue().isPresent());
+ }
+
+ @Test
+ public void testCommitSyncWithError()
+ {
+ final TopicIdPartition tip = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("test-topic", 0));
+ final KafkaException error = new KafkaException("commit failed");
+ final Map<TopicIdPartition, Optional<KafkaException>> kafkaResult = new HashMap<>();
+ kafkaResult.put(tip, Optional.of(error));
+
+ when(mockConsumer.commitSync()).thenReturn(kafkaResult);
+
+ final Map<KafkaTopicPartition, Optional<Exception>> result = supplier.commitSync();
+ Assert.assertEquals(1, result.size());
+
+ final Optional<Exception> maybeError = result.values().iterator().next();
+ Assert.assertTrue(maybeError.isPresent());
+ Assert.assertEquals("commit failed", maybeError.get().getMessage());
+ }
+
+ @Test
+ public void testCloseIsIdempotent()
+ {
+ supplier.close();
+ supplier.close();
+ verify(mockConsumer, Mockito.times(1)).close();
+ }
+
+ @Test
+ public void testWakeupForwardsToConsumer()
+ {
+ supplier.wakeup();
+ Mockito.verify(mockConsumer).wakeup();
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsForwardsToConsumer()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.of(45_000));
+ final Optional<Integer> lockMs = supplier.acquisitionLockTimeoutMs();
+ Assert.assertTrue(lockMs.isPresent());
+ Assert.assertEquals(Integer.valueOf(45_000), lockMs.get());
+ }
+
+ @Test
+ public void testAcknowledgeTwoArgOverloadDefaultsToAccept() throws Exception
+ {
+ final String topic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ topic,
+ 0,
+ 100L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(topic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ supplier.poll(1000L);
+ supplier.acknowledge(new KafkaTopicPartition(true, topic, 0), 100L);
+
+ verify(mockConsumer).acknowledge(record, org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT);
+ }
+
+ @Test
+ public void testGetPartitionIdsReturnsEmptySet()
+ {
+ Assert.assertEquals(Set.of(), supplier.getPartitionIds("any-stream"));
+ }
+
+ @Test
+ public void testAcquisitionLockTimeoutMsEmpty()
+ {
+ Mockito.when(mockConsumer.acquisitionLockTimeoutMs()).thenReturn(Optional.empty());
+ Assert.assertTrue(supplier.acquisitionLockTimeoutMs().isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeWithRenewType()
+ {
+ final String testTopic = "test-topic";
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ testTopic,
+ 0,
+ 200L,
+ "key".getBytes(StandardCharsets.UTF_8),
+ "value".getBytes(StandardCharsets.UTF_8)
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(testTopic, 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11248)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3210682966
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = Mockito.mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ Mockito.when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ Mockito.verify(mockConsumer).subscribe(topics);
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ Mockito.verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11185)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = Mockito.mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ Mockito.when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ Mockito.verify(mockConsumer).subscribe(topics);
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ Mockito.verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assert.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11187)
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/common/AcknowledgingRecordSupplier.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.seekablestream.common;
+
+import org.apache.druid.data.input.impl.ByteEntity;
+
+import javax.validation.constraints.NotNull;
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Record supplier for queue-semantics streams (e.g. Kafka share groups,
+ * KIP-932) where the broker owns delivery state and consumers acknowledge
+ * records explicitly. Unlike {@link RecordSupplier}, callers do not assign
+ * or seek partitions; they subscribe to topics and ack individual records.
+ *
+ * @param <PartitionIdType> partition identifier type
+ * @param <SequenceOffsetType> sequence/offset number type
+ * @param <RecordType> record entity type
+ */
+public interface AcknowledgingRecordSupplier<PartitionIdType, SequenceOffsetType, RecordType extends ByteEntity>
+ extends Closeable
+{
+ void subscribe(Set<String> topics);
+
+ void unsubscribe();
+
+ Set<String> subscription();
+
+ /**
+ * Poll for records. Records carry acquisition locks and must be
+ * acknowledged before the lock expires.
+ */
+ @NotNull
+ List<OrderedPartitionableRecord<PartitionIdType, SequenceOffsetType, RecordType>> poll(long timeoutMs);
+
+ /** Acknowledge a single record with the default type ({@code ACCEPT}). */
+ void acknowledge(PartitionIdType partitionId, SequenceOffsetType offset);
+
+ /** Acknowledge a single record with the given type. */
+ void acknowledge(PartitionIdType partitionId, SequenceOffsetType offset, AcknowledgeType type);
+
+ /** Acknowledge a batch of records with the given type. */
+ void acknowledge(Map<PartitionIdType, Collection<SequenceOffsetType>> offsets, AcknowledgeType type);
+
+ /**
+ * Commit pending acknowledgements; returns a per-partition exception when
+ * the commit failed for that partition.
+ */
+ Map<PartitionIdType, Optional<Exception>> commitSync();
+
+ Set<PartitionIdType> getPartitionIds(String stream);
Review Comment:
## CodeQL / Useless parameter
The parameter 'stream' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/11188)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @Before
+ public void setUp()
+ {
+ mockConsumer = Mockito.mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @After
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ Mockito.when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ Mockito.verify(mockConsumer).subscribe(topics);
+ Assert.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ Mockito.verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assert.assertEquals(2, result.size());
+
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assert.assertNotNull(polled1);
+ Assert.assertEquals("test-topic", polled1.getStream());
+ Assert.assertEquals(0, polled1.getPartitionId().partition());
+ Assert.assertNotNull(polled1.getData());
+ Assert.assertEquals(1, polled1.getData().size());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assert.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/11186)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3212557778
##########
extensions-core/kafka-indexing-service/pom.xml:
##########
@@ -252,6 +257,30 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <!--
+ The bundled in-JVM test broker (kafka_2.13:3.9.x, ZooKeeper-mode) was
+ compiled against kafka-clients 3.9.x and references Utils.mkSet(Object[]),
+ which was removed in kafka-clients 4.x. This module pins kafka-clients to
+ 4.2.0 to enable KIP-932 share-group support, so KafkaConfig static
+ initialization throws NoSuchMethodError under any JVM. The tests below
+ depend on TestBroker and must be skipped until the test broker migrates to
+ kafka_2.13:4.x (KRaft) or Testcontainers. Share-group code paths are
+ covered by ShareGroupIndexTaskRunnerTest, KafkaShareGroupRecordSupplierTest
+ (mock-based) and embedded-tests/EmbeddedShareGroupIngestionTest
+ (Testcontainers). Tracked as a follow-up of apache/druid#19311.
+ -->
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <exclude>**/KafkaIndexTaskTest.java</exclude>
Review Comment:
Yes, planning to fix (upgrade to 4.2 version in all places) as part of this https://github.com/apache/druid/issues/19322 - I feel that we need bunch of changes, API compatibility issues in running tests , kafka docker image for IT tests and in source code - so want to keep it in separate PR.
This PR mainly focus on Share group implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3242936105
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplier.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+
+import javax.annotation.Nonnull;
+import javax.validation.constraints.NotNull;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Adapts {@link KafkaShareConsumer} to {@link AcknowledgingRecordSupplier}.
+ * Delivery state lives on the broker; the supplier sets {@code group.id} to
+ * the configured share group name.
+ */
+public class KafkaShareGroupRecordSupplier
+ implements AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>
+{
+ private static final Logger log = new Logger(KafkaShareGroupRecordSupplier.class);
+
+ private final KafkaShareConsumer<byte[], byte[]> consumer;
+
+ /**
+ * Records returned by the most recent {@link #poll(long)}, retained so
+ * {@link #acknowledge} can pass the original {@link ConsumerRecord} to the
+ * Kafka client. The {@code (topic, partition, offset)} ack variant is fragile
+ * once the share-fetch buffer rolls over (KIP-932).
+ */
+ private final Map<RecordKey, ConsumerRecord<byte[], byte[]>> deliveredRecords = new HashMap<>();
+ private boolean closed;
+
+ public KafkaShareGroupRecordSupplier(
+ Map<String, Object> consumerProperties,
+ ObjectMapper sortingMapper,
+ String groupId
+ )
+ {
+ this(createShareConsumer(consumerProperties, sortingMapper, groupId));
+ }
+
+ @VisibleForTesting
+ public KafkaShareGroupRecordSupplier(KafkaShareConsumer<byte[], byte[]> consumer)
+ {
+ this.consumer = consumer;
+ }
+
+ @Override
+ public void subscribe(Set<String> topics)
+ {
+ consumer.subscribe(topics);
+ }
+
+ @Override
+ public void unsubscribe()
+ {
+ consumer.unsubscribe();
+ }
+
+ @Override
+ public Set<String> subscription()
+ {
+ return consumer.subscription();
+ }
+
+ @NotNull
+ @Override
+ public List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> poll(long timeoutMs)
+ {
+ deliveredRecords.clear();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> polledRecords =
+ new ArrayList<>();
+ for (ConsumerRecord<byte[], byte[]> record : consumer.poll(Duration.ofMillis(timeoutMs))) {
+ deliveredRecords.put(new RecordKey(record.topic(), record.partition(), record.offset()), record);
+ polledRecords.add(new OrderedPartitionableRecord<>(
+ record.topic(),
+ new KafkaTopicPartition(true, record.topic(), record.partition()),
+ record.offset(),
+ record.value() == null ? null : ImmutableList.of(new KafkaRecordEntity(record)),
+ record.timestamp()
+ ));
+ }
+ return polledRecords;
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset)
+ {
+ acknowledge(partitionId, offset, AcknowledgeType.ACCEPT);
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset, AcknowledgeType type)
+ {
+ final String topic = partitionId.topic().orElseThrow(
+ () -> new IllegalArgumentException("Cannot acknowledge record without topic")
+ );
+ final ConsumerRecord<byte[], byte[]> record = deliveredRecords.get(
+ new RecordKey(topic, partitionId.partition(), offset)
+ );
+ if (record == null) {
+ throw new IllegalStateException(StringUtils.format(
+ "Cannot acknowledge unknown record at topic[%s] partition[%d] offset[%d]; "
+ + "either it was not delivered by the most recent poll() or it has already been acknowledged.",
+ topic, partitionId.partition(), offset
+ ));
+ }
+ consumer.acknowledge(record, toKafkaAcknowledgeType(type));
+ }
+
+ @Override
+ public void acknowledge(
+ Map<KafkaTopicPartition, Collection<Long>> offsets,
+ AcknowledgeType type
+ )
+ {
+ final org.apache.kafka.clients.consumer.AcknowledgeType kafkaType = toKafkaAcknowledgeType(type);
+ for (Map.Entry<KafkaTopicPartition, Collection<Long>> entry : offsets.entrySet()) {
+ final KafkaTopicPartition partition = entry.getKey();
+ final String topic = partition.topic().orElseThrow(
+ () -> new IllegalArgumentException("Cannot acknowledge record without topic")
+ );
+ for (Long offset : entry.getValue()) {
+ final ConsumerRecord<byte[], byte[]> record = deliveredRecords.get(
+ new RecordKey(topic, partition.partition(), offset)
+ );
+ if (record == null) {
+ throw new IllegalStateException(StringUtils.format(
+ "Cannot acknowledge unknown record at topic[%s] partition[%d] offset[%d]; "
+ + "either it was not delivered by the most recent poll() or it has already been acknowledged.",
+ topic, partition.partition(), offset
+ ));
+ }
+ consumer.acknowledge(record, kafkaType);
+ }
+ }
+ }
+
+ @Override
+ public Map<KafkaTopicPartition, Optional<Exception>> commitSync()
+ {
+ final Map<TopicIdPartition, Optional<KafkaException>> result = consumer.commitSync();
+ final Map<KafkaTopicPartition, Optional<Exception>> mapped = new HashMap<>();
+ for (Map.Entry<TopicIdPartition, Optional<KafkaException>> entry : result.entrySet()) {
+ final TopicIdPartition tip = entry.getKey();
+ mapped.put(
+ new KafkaTopicPartition(true, tip.topic(), tip.partition()),
+ entry.getValue().map(e -> (Exception) e)
+ );
+ }
+ return mapped;
+ }
+
+ @Override
+ public Set<KafkaTopicPartition> getPartitionIds(String stream)
+ {
+ // Share consumer does not expose partition assignment; broker manages it.
+ return Collections.emptySet();
+ }
+
+ @Override
+ public void wakeup()
+ {
+ consumer.wakeup();
+ }
+
+ @Override
+ public Optional<Integer> acquisitionLockTimeoutMs()
+ {
+ return consumer.acquisitionLockTimeoutMs();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ try {
+ consumer.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing KafkaShareConsumer");
+ }
+ }
+
+ @Nonnull
+ private static org.apache.kafka.clients.consumer.AcknowledgeType toKafkaAcknowledgeType(AcknowledgeType type)
+ {
+ switch (type) {
+ case ACCEPT:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT;
+ case RELEASE:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE;
+ case REJECT:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.REJECT;
+ case RENEW:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.RENEW;
+ default:
+ throw new IllegalArgumentException("Unknown acknowledge type: " + type);
+ }
+ }
+
+ private static KafkaShareConsumer<byte[], byte[]> createShareConsumer(
+ Map<String, Object> consumerProperties,
+ ObjectMapper sortingMapper,
+ String groupId
+ )
+ {
+ final Map<String, Object> sanitized = ShareGroupConsumerProperties.sanitize(consumerProperties);
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3242791337
##########
docs/ingestion/kafka-share-group-ingestion.md:
##########
@@ -0,0 +1,337 @@
+---
+id: kafka-share-group-ingestion
+title: "Kafka share group ingestion"
+sidebar_label: "Kafka share group ingestion"
+description: "Queue-semantics ingestion from Apache Kafka using share groups (KIP-932). Scale consumers beyond partition count with at-least-once delivery."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+Requires Apache Kafka 4.0 or higher with share groups (KIP-932) enabled on the broker.
+:::
+
+## Overview
+
+Kafka share groups (KIP-932) let multiple consumers read from the same partition concurrently. The broker manages per-record acquisition locks and explicit acknowledgement, so consumer count is not capped by partition count, joining or leaving consumers does not pause the group, and a slow record does not block its partition.
+
+Druid's `ShareGroupIndexTask` consumes from a share group and publishes segments with at-least-once delivery: records are acknowledged only after their segments are atomically registered in the metadata store.
+
+## When to use share group ingestion
+
+| Scenario | Consumer group | Share group |
+|----------|---------------|-------------|
+| Workers needed exceed partition count | Idle workers | All workers active |
+| Elastic scaling (auto-scale events) | Rebalancing pause (30-60s) | Zero pause |
+| Per-message processing time varies | Head-of-line blocking | Independent processing |
+| Ordered processing required per partition | Yes | No (delivery order not guaranteed) |
+
+Choose share groups when throughput and elastic scaling matter more than strict per-partition ordering.
+
+## Task spec
+
+Submit a `ShareGroupIndexTask` to the Overlord. There are no start/end offsets -- the broker tracks them.
+
+```json
+{
+ "type": "index_kafka_share_group",
+ "dataSchema": {
+ "dataSource": "my_datasource",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "useSchemaDiscovery": true
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": "NONE"
+ }
+ },
+ "ioConfig": {
+ "topic": "my_topic",
+ "groupId": "druid-share-group",
+ "consumerProperties": {
+ "bootstrap.servers": "kafka-broker:9092"
+ },
+ "inputFormat": {
+ "type": "json"
+ },
+ "pollTimeout": 2000
+ },
+ "tuningConfig": {
+ "type": "KafkaTuningConfig",
+ "maxRowsPerSegment": 5000000
+ }
+}
+```
+
+## IO configuration
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `topic` | String | Yes | -- | Kafka topic to consume from. |
+| `groupId` | String | Yes | -- | Share group identifier. Multiple tasks with the same `groupId` share the workload. |
+| `consumerProperties` | Map | Yes | -- | Kafka consumer properties. Must include `bootstrap.servers`. See [Consumer property restrictions](#consumer-property-restrictions). |
+| `inputFormat` | Object | Yes | -- | Input format for parsing records (json, csv, avro, etc.). |
+| `pollTimeout` | Long | No | 2000 | Poll timeout in milliseconds. |
+
+### Consumer property restrictions
+
+Share consumers (KIP-932) reject some keys that are valid for regular consumer groups. Druid silently strips the keys below from `consumerProperties` (with a `WARN` log per stripped key) before constructing the `KafkaShareConsumer`:
+
+| Stripped key | Why |
+|--------------|-----|
+| `auto.offset.reset` | Initial position is broker-controlled for share groups. |
+| `enable.auto.commit` | Share consumers always require explicit `acknowledge()` + `commitSync()`. |
+| `group.instance.id` | Share groups do not support static membership. |
+| `isolation.level` | Always read-committed for share groups. |
+| `partition.assignment.strategy` | Broker controls per-record delivery for share groups. |
+| `interceptor.classes` | Not supported for share consumers. |
+| `session.timeout.ms` | Share groups have no consumer-group session model. |
+| `heartbeat.interval.ms` | Share groups have no heartbeat. |
+| `group.protocol` | Always `SHARE` for share consumers. |
+| `group.remote.assignor` | Not applicable to share groups. |
+
+`share.acknowledgement.mode=explicit` is set automatically and must not be overridden.
+
+### Tuning configuration
+
+`tuningConfig` accepts the standard `KafkaTuningConfig` fields. The runner currently honors:
+
+- `maxRowsInMemory` / `maxBytesInMemory`: triggers a mid-batch persist when the appenderator signals `isPersistRequired`.
+- `maxRowsPerSegment`: when reached during a batch the runner logs the event; over-threshold segments are pushed at the end-of-batch publish boundary.
+
+Mid-batch checkpoint and sequence rollover are not supported.
+
+## How it works
+
+1. The task subscribes to the topic with a `KafkaShareConsumer` using the configured `groupId`.
+2. The broker delivers batches of records with per-record acquisition locks.
+3. Each polled record is parsed by `StreamChunkReader` (the same multi-row parser as `KafkaIndexTask`); a record may produce zero, one, or many `InputRow`s. All resulting rows are added to the appenderator before the record is acknowledged.
+4. Parse failures go through `ParseExceptionHandler` (so `maxParseExceptions` is honored). Bytes/processed/unparseable counters are incremented exactly once per row.
+5. Segments persist mid-batch on memory pressure and unconditionally at end-of-batch, then publish atomically via `SegmentTransactionalAppendAction`.
+6. After a successful publish, every offset in the batch is acknowledged with `ACCEPT` and a `commitSync()` flushes acknowledgements to the broker.
+7. On task failure or graceful stop before publish, unacknowledged records are redelivered by the broker after the acquisition lock expires.
+
+## Safety invariants
+
+1. **ACK after publish:** `ACCEPT` is sent only after the segment is registered in the metadata store. No data loss on task failure.
+2. **Multi-row safe:** every row produced from a record is added to the appenderator before that record is acknowledged.
+3. **Resource safe:** `Appenderator` and `KafkaShareConsumer` are released on every exit path.
+4. **Terminal state:** every polled record reaches exactly one terminal state -- `ACCEPT`, `RELEASE`, or broker redelivery after lock expiry.
+
+## Graceful stop
+
+When the Overlord asks a task to stop, the runner calls `KafkaShareConsumer.wakeup()`. The in-flight `poll()` throws `WakeupException`; the runner exits the loop after committing any in-flight batch. Records polled but not yet published remain unacknowledged and are redelivered by the broker after the acquisition lock expires.
+
+## Acquisition lock duration
+
+The broker controls the lock via `group.share.record.lock.duration.ms`. The runner logs the effective value once after the first poll:
+
+```
+Effective broker acquisition lock timeout for share-group[my-group]: 30000 ms
+```
+
+A single thread does both poll and publish. If a batch exceeds the lock duration, in-flight records may be redelivered (duplicates). Tune `pollTimeout`, `maxRowsInMemory`, and `maxRowsPerSegment` so each cycle stays well under the lock window.
+
+## Scaling
+
+Tasks with the same `groupId` share the workload automatically; you can run more tasks than partitions:
+
+```
+Topic: 4 partitions
+Tasks with same groupId: 20
+Result: All 20 tasks actively consuming (broker distributes records)
+```
+
+Adding or removing tasks does not trigger a rebalancing pause.
+
+## Delivery semantics
+
+At-least-once. On task failure, records between the last committed acknowledgement and the failure point are redelivered, which may produce duplicates across restarts. A deduplication cache is planned.
+
+## Metrics
+
+In addition to the standard ingestion metrics (`ingest/events/processed`, `ingest/events/unparseable`, `ingest/persists/count`, etc.), share-group ingestion emits:
+
+| Metric | Description |
+|--------|-------------|
+| `ingest/shareGroup/commitFailures` | Per-batch count of partitions whose `commitSync()` failed. A non-zero value means the affected records will be redelivered; alert on sustained non-zero values. |
+
+## Limitations (current release)
+
+- Single-threaded ingestion per task; a future enhancement may add a background `RENEW` thread to extend the broker lock for long-running batches.
+- No supervisor integration; tasks are submitted manually via the Overlord API. A `KafkaShareGroupSupervisor` is planned as a future enhancement.
+- No deduplication cache (at-least-once).
+- Delivery order within a partition is not guaranteed.
+- Mid-batch checkpoint / sequence rollover is not supported. If a batch grossly exceeds `maxRowsPerSegment` the runner still publishes correctly (multiple segments per batch), but the threshold is only checked at end-of-batch boundaries.
+
+## Demo: end-to-end validation with Druid UI
+
+### Prerequisites
+
+- Java 17
+- Kafka 4.2.0 (with share groups enabled)
+- Druid checked out from this repository (built from source)
+
+### Step 1: Start Kafka with share groups
+
+```bash
+cd kafka_2.13-4.2.0
+
+KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
+bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
+
+echo "group.share.enable=true" >> config/server.properties
+echo "group.share.record.lock.duration.ms=30000" >> config/server.properties
+
+bin/kafka-server-start.sh config/server.properties
+```
+
+### Step 2: Create topic and produce messages
+
+```bash
+cd kafka_2.13-4.2.0
+
+bin/kafka-topics.sh --create --topic druid-share-test --partitions 4 --bootstrap-server localhost:9092
+
+bin/kafka-console-producer.sh --topic druid-share-test --bootstrap-server localhost:9092
Review Comment:
Updated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4124329095
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupBatchStore.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.storage.StorageConnector;
+
+import java.io.FilterOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.DigestInputStream;
+import java.security.DigestOutputStream;
+import java.security.MessageDigest;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.UUID;
+
+final class ShareGroupBatchStore
+{
+ static final class StoredBatch
+ {
+ private final String path;
+ private final String sha256;
+ private final long size;
+ private final ShareGroupRawBatch batch;
+
+ private StoredBatch(String path, String sha256, long size, ShareGroupRawBatch batch)
+ {
+ this.path = path;
+ this.sha256 = sha256;
+ this.size = size;
+ this.batch = batch;
+ }
+
+ String getPath()
+ {
+ return path;
+ }
+
+ String getSha256()
+ {
+ return sha256;
+ }
+
+ long getSize()
+ {
+ return size;
+ }
+
+ ShareGroupRawBatch getBatch()
+ {
+ return batch;
+ }
+ }
+
+ private static final class CountingOutputStream extends FilterOutputStream
+ {
+ private long count;
+
+ private CountingOutputStream(OutputStream outputStream)
+ {
+ super(outputStream);
+ }
+
+ @Override
+ public void write(int value) throws IOException
+ {
+ out.write(value);
+ count++;
+ }
+
+ @Override
+ public void write(byte[] bytes, int offset, int length) throws IOException
+ {
+ out.write(bytes, offset, length);
+ count += length;
+ }
+
+ private long getCount()
+ {
+ return count;
+ }
+ }
+
+ private final StorageConnector storageConnector;
+ private final String taskId;
+ private final ShareGroupRawBatchWriter writer = new ShareGroupRawBatchWriter();
+ private final ShareGroupRawBatchReader reader = new ShareGroupRawBatchReader();
+
+ ShareGroupBatchStore(StorageConnector storageConnector, String taskId)
+ {
+ this.storageConnector = Preconditions.checkNotNull(storageConnector, "storageConnector");
+ this.taskId = Preconditions.checkNotNull(taskId, "taskId");
+ }
+
+ StoredBatch store(
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records
+ ) throws IOException
+ {
+ return store(ShareGroupRawBatch.fromOrderedRecords(records));
+ }
+
+ StoredBatch store(ShareGroupRawBatch batch) throws IOException
+ {
+ final String path = objectPath(batch);
+ final MessageDigest writeDigest = ShareGroupRawBatchWriter.newDigest();
+ final CountingOutputStream countingOutput;
+ try (OutputStream storageOutput = storageConnector.write(path)) {
+ countingOutput = new CountingOutputStream(storageOutput);
+ writer.write(batch, new DigestOutputStream(countingOutput, writeDigest));
Review Comment:
## CodeQL / Potential output resource leak
This DigestOutputStream is not always closed on method exit.
[Show more details](https://github.com/apache/druid/security/code-scanning/12022)
##########
server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java:
##########
@@ -4849,6 +5225,135 @@
.build();
}
+ private static ShareInboxBatch shareInboxBatch(String manifestId, String fingerprint, List<Long> offsets)
+ {
+ return new ShareInboxBatch(
+ "datasource",
+ "inbox",
+ "group",
+ "cluster",
+ "topic-id",
+ "topic",
+ 0,
+ fingerprint,
+ manifestId,
+ "path/" + manifestId,
+ "hash-" + manifestId,
+ 100,
+ offsets,
+ 4_096
+ );
+ }
+
+ private static ShareInboxClaimRequest shareInboxClaimRequest(String owner, long maxBytes)
+ {
+ return new ShareInboxClaimRequest(
+ "datasource",
+ "inbox",
+ "fingerprint",
+ owner,
+ 10,
+ 10,
+ maxBytes,
+ 60_000
+ );
+ }
+
+ private static ShareInboxRenewRequest shareInboxRenewRequest(String owner, Map<String, Long> claims)
+ {
+ return new ShareInboxRenewRequest(
+ "datasource",
+ "inbox",
+ "fingerprint",
+ owner,
+ claims,
+ 60_000
+ );
+ }
+
+ private static ShareInboxCompletionRequest shareInboxCompletionRequest(
+ ShareInboxManifest manifest,
+ String completionId
+ )
+ {
+ return new ShareInboxCompletionRequest(
+ manifest.getDataSource(),
+ manifest.getInboxId(),
+ manifest.getSpecFingerprint(),
+ manifest.getClaimOwner(),
+ Map.of(manifest.getManifestId(), manifest.getClaimEpoch()),
+ completionId
+ );
+ }
+
+ private static DataSegment shareInboxSegment(String version)
+ {
+ return new DataSegment(
+ "datasource",
+ Intervals.of("2026-01-01/2026-01-02"),
+ version,
+ Map.of(),
+ List.of("dim"),
+ List.of("metric"),
+ new LinearShardSpec(0),
+ 9,
+ 100
+ );
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [DataSegment.DataSegment](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/12031)
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/StageShareInboxBatchAction.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.ShareInboxBatch;
+import org.apache.druid.indexing.overlord.ShareInboxStageResult;
+
+import java.util.Objects;
+
+public class StageShareInboxBatchAction implements TaskAction<ShareInboxStageResult>
+{
+ public static final String TYPE = "stageShareInboxBatch";
+
+ private final ShareInboxBatch batch;
+
+ @JsonCreator
+ public StageShareInboxBatchAction(@JsonProperty("batch") ShareInboxBatch batch)
+ {
+ this.batch = Objects.requireNonNull(batch, "batch");
+ }
+
+ @JsonProperty
+ public ShareInboxBatch getBatch()
+ {
+ return batch;
+ }
+
+ @Override
+ public TypeReference<ShareInboxStageResult> getReturnTypeReference()
+ {
+ return new TypeReference<>() {};
+ }
+
+ @Override
+ public ShareInboxStageResult perform(Task task, TaskActionToolbox toolbox)
+ {
+ if (!task.getDataSource().equals(batch.getDataSource())) {
+ throw InvalidInput.exception(
+ "Task[%s] cannot stage share inbox records for datasource[%s]",
+ task.getId(),
+ batch.getDataSource()
+ );
+ }
+ return toolbox.getIndexerMetadataStorageCoordinator().stageShareInboxBatch(batch);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "StageShareInboxBatchAction{" +
+ "batch=" + batch +
Review Comment:
## CodeQL / Use of default toString()
Default toString(): ShareInboxBatch inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/druid/security/code-scanning/12028)
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentTransactionalAppendFromShareInboxAction.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.PendingSegmentAllocatingTask;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.CriticalAction;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.overlord.ShareInboxCompletionRequest;
+import org.apache.druid.metadata.ReplaceTaskLock;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.timeline.DataSegment;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class SegmentTransactionalAppendFromShareInboxAction implements TaskAction<SegmentPublishResult>
+{
+ public static final String TYPE = "segmentTransactionalAppendFromShareInbox";
+
+ private final Set<DataSegment> segments;
+ private final ShareInboxCompletionRequest completionRequest;
+ @Nullable
+ private final SegmentSchemaMapping segmentSchemaMapping;
+
+ @JsonCreator
+ public SegmentTransactionalAppendFromShareInboxAction(
+ @JsonProperty("segments") Set<DataSegment> segments,
+ @JsonProperty("completionRequest") ShareInboxCompletionRequest completionRequest,
+ @JsonProperty("segmentSchemaMapping") @Nullable SegmentSchemaMapping segmentSchemaMapping
+ )
+ {
+ this.segments = Set.copyOf(Objects.requireNonNull(segments, "segments"));
+ this.completionRequest = Objects.requireNonNull(completionRequest, "completionRequest");
+ this.segmentSchemaMapping = segmentSchemaMapping;
+ }
+
+ @JsonProperty
+ public Set<DataSegment> getSegments()
+ {
+ return segments;
+ }
+
+ @JsonProperty
+ public ShareInboxCompletionRequest getCompletionRequest()
+ {
+ return completionRequest;
+ }
+
+ @Nullable
+ @JsonProperty
+ public SegmentSchemaMapping getSegmentSchemaMapping()
+ {
+ return segmentSchemaMapping;
+ }
+
+ @Override
+ public TypeReference<SegmentPublishResult> getReturnTypeReference()
+ {
+ return new TypeReference<>() {};
+ }
+
+ @Override
+ public SegmentPublishResult perform(Task task, TaskActionToolbox toolbox)
+ {
+ if (!(task instanceof PendingSegmentAllocatingTask)) {
+ throw DruidException.defensive(
+ "Task[%s] of type[%s] cannot append share inbox segments because it does not implement PendingSegmentAllocatingTask.",
+ task.getId(),
+ task.getType()
+ );
+ }
+ ClaimShareInboxManifestsAction.validateTask(
+ task,
+ completionRequest.getDataSource(),
+ completionRequest.getTaskId()
+ );
+
+ final List<TaskLock> locks = toolbox.getTaskLockbox().findLocksForTask(task);
+ for (TaskLock lock : locks) {
+ if (lock.getType() != TaskLockType.APPEND) {
+ throw InvalidInput.exception(
+ "Cannot use action[%s] for task[%s] as it is holding a lock of type[%s] instead of [APPEND].",
+ TYPE,
+ task.getId(),
+ lock.getType()
+ );
+ }
+ }
+
+ final PendingSegmentAllocatingTask allocatingTask = (PendingSegmentAllocatingTask) task;
+ final SegmentPublishResult result;
+ if (segments.isEmpty()) {
+ result = commit(toolbox, allocatingTask.getTaskAllocatorId(), Map.of());
+ } else {
+ TaskLocks.checkLockCoversSegments(task, toolbox.getTaskLockbox(), segments);
+ final Map<DataSegment, ReplaceTaskLock> segmentToReplaceLock = TaskLocks.findReplaceLocksCoveringSegments(
+ task.getDataSource(),
+ toolbox.getTaskLockbox(),
+ segments
+ );
+ try {
+ result = toolbox.getTaskLockbox().doInCriticalSection(
+ task,
+ segments.stream().map(DataSegment::getInterval).collect(Collectors.toSet()),
+ CriticalAction.<SegmentPublishResult>builder()
+ .onValidLocks(() -> commit(toolbox, allocatingTask.getTaskAllocatorId(), segmentToReplaceLock))
+ .onInvalidLocks(
+ () -> SegmentPublishResult.fail(
+ "Invalid task locks. Maybe they are revoked by a higher priority task."
+ + " Please check the overlord log for details."
+ )
+ )
+ .build()
+ );
+ }
+ catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ IndexTaskUtils.emitSegmentPublishMetrics(result, task, toolbox);
+ return result;
+ }
+
+ private SegmentPublishResult commit(
+ TaskActionToolbox toolbox,
+ String taskAllocatorId,
+ Map<DataSegment, ReplaceTaskLock> segmentToReplaceLock
+ )
+ {
+ return toolbox.getIndexerMetadataStorageCoordinator().commitAppendSegmentsAndShareInbox(
+ segments,
+ segmentToReplaceLock,
+ taskAllocatorId,
+ segmentSchemaMapping,
+ completionRequest
+ );
+ }
+
+ @Override
+ public String toString()
+ {
+ return "SegmentTransactionalAppendFromShareInboxAction{" +
+ "segments=" + segments +
+ ", completionRequest=" + completionRequest +
Review Comment:
## CodeQL / Use of default toString()
Default toString(): ShareInboxCompletionRequest inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/druid/security/code-scanning/12027)
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/ClaimShareInboxManifestsAction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.ShareInboxClaimRequest;
+import org.apache.druid.indexing.overlord.ShareInboxClaimResult;
+
+import java.util.Objects;
+
+public class ClaimShareInboxManifestsAction implements TaskAction<ShareInboxClaimResult>
+{
+ public static final String TYPE = "claimShareInboxManifests";
+
+ private final ShareInboxClaimRequest request;
+
+ @JsonCreator
+ public ClaimShareInboxManifestsAction(@JsonProperty("request") ShareInboxClaimRequest request)
+ {
+ this.request = Objects.requireNonNull(request, "request");
+ }
+
+ @JsonProperty
+ public ShareInboxClaimRequest getRequest()
+ {
+ return request;
+ }
+
+ @Override
+ public TypeReference<ShareInboxClaimResult> getReturnTypeReference()
+ {
+ return new TypeReference<>() {};
+ }
+
+ @Override
+ public ShareInboxClaimResult perform(Task task, TaskActionToolbox toolbox)
+ {
+ validateTask(task, request.getDataSource(), request.getClaimOwner());
+ return toolbox.getIndexerMetadataStorageCoordinator().claimShareInboxManifests(request);
+ }
+
+ static void validateTask(Task task, String dataSource, String claimOwner)
+ {
+ if (!task.getDataSource().equals(dataSource) || !task.getId().equals(claimOwner)) {
+ throw InvalidInput.exception(
+ "Task[%s] cannot manage share inbox claims for datasource[%s] as owner[%s]",
+ task.getId(),
+ dataSource,
+ claimOwner
+ );
+ }
+ }
+
+ @Override
+ public String toString()
+ {
+ return "ClaimShareInboxManifestsAction{" +
+ "request=" + request +
Review Comment:
## CodeQL / Use of default toString()
Default toString(): ShareInboxClaimRequest inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/druid/security/code-scanning/12025)
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupAcquisitionLoop.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.common.errors.WakeupException;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executor;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.ToLongFunction;
+
+final class ShareGroupAcquisitionLoop
+{
+ private final String topic;
+ private final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier;
+ private final ShareGroupBatchStager batchStager;
+ private final Executor stagingExecutor;
+ private final ShareGroupAcquisitionRegistry registry;
+ private final ToLongFunction<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> byteEstimator;
+ private final long pollTimeoutMs;
+ private final long maxPollIntervalMs;
+ private final BlockingQueue<ShareGroupStagingCompletion> completions = new LinkedBlockingQueue<>();
+ private final AtomicBoolean stopRequested = new AtomicBoolean();
+ private Thread ownerThread;
+
+ ShareGroupAcquisitionLoop(
+ String topic,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ ShareGroupBatchStager batchStager,
+ Executor stagingExecutor,
+ ShareGroupAcquisitionRegistry registry,
+ ToLongFunction<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> byteEstimator,
+ long pollTimeoutMs,
+ long maxPollIntervalMs
+ )
+ {
+ Preconditions.checkArgument(pollTimeoutMs >= 0, "pollTimeoutMs must not be negative");
+ Preconditions.checkArgument(maxPollIntervalMs > 0, "maxPollIntervalMs must be positive");
+ this.topic = Preconditions.checkNotNull(topic, "topic");
+ this.recordSupplier = Preconditions.checkNotNull(recordSupplier, "recordSupplier");
+ this.batchStager = Preconditions.checkNotNull(batchStager, "batchStager");
+ this.stagingExecutor = Preconditions.checkNotNull(stagingExecutor, "stagingExecutor");
+ this.registry = Preconditions.checkNotNull(registry, "registry");
+ this.byteEstimator = Preconditions.checkNotNull(byteEstimator, "byteEstimator");
+ this.pollTimeoutMs = pollTimeoutMs;
+ this.maxPollIntervalMs = maxPollIntervalMs;
+ }
+
+ void run() throws Exception
+ {
+ checkOwnerThread();
+ recordSupplier.subscribe(Set.of(topic));
+ while (!stopRequested.get()) {
+ try {
+ runOnce();
+ }
+ catch (WakeupException e) {
+ if (!stopRequested.get()) {
+ throw e;
+ }
+ }
+ }
+ }
+
+ void runOnce() throws InterruptedException
+ {
+ checkOwnerThread();
+ drainCompletions();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(pollTimeoutMs);
+ drainCompletions();
+ if (records.isEmpty()) {
+ return;
+ }
+
+ final long acquisitionLockTimeoutMs = recordSupplier.acquisitionLockTimeoutMs().orElse(-1);
+ final Map<ShareGroupRecordIdentity,
+ OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> tracked = new LinkedHashMap<>();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> saturated =
+ new ArrayList<>();
+ final Map<KafkaTopicPartition,
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>>> newRecordsByPartition =
+ new LinkedHashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ final ShareGroupRecordIdentity identity = ShareGroupRecordIdentity.from(record);
+ final ShareGroupAcquisitionRegistry.Observation observation = registry.observe(
+ record,
+ byteEstimator.applyAsLong(record),
+ acquisitionLockTimeoutMs,
+ maxPollIntervalMs
+ );
+ if (observation == ShareGroupAcquisitionRegistry.Observation.SATURATED) {
+ saturated.add(record);
+ } else {
+ tracked.put(identity, record);
+ if (observation == ShareGroupAcquisitionRegistry.Observation.ADDED) {
+ newRecordsByPartition.computeIfAbsent(record.getPartitionId(), ignored -> new ArrayList<>()).add(record);
+ }
+ }
+ }
+
+ for (List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> partitionRecords
+ : newRecordsByPartition.values()) {
+ submit(partitionRecords);
+ }
+
+ waitForAcknowledgementDecisions(tracked.keySet());
+
+ boolean terminalAcknowledgementAssigned = false;
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : saturated) {
+ recordSupplier.acknowledge(
+ record.getPartitionId(),
+ record.getSequenceNumber(),
+ AcknowledgeType.RELEASE
+ );
+ terminalAcknowledgementAssigned = true;
+ }
+ for (Map.Entry<ShareGroupRecordIdentity,
+ OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> trackedRecord : tracked.entrySet()) {
+ final AcknowledgeType type = registry.acknowledgementFor(
+ trackedRecord.getKey(),
+ stopRequested.get()
+ ).orElseThrow(() -> new IllegalStateException("No acknowledgement decision for " + trackedRecord.getKey()));
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record = trackedRecord.getValue();
+ recordSupplier.acknowledge(record.getPartitionId(), record.getSequenceNumber(), type);
+ registry.acknowledgementAssigned(trackedRecord.getKey(), type);
+ terminalAcknowledgementAssigned |= type != AcknowledgeType.RENEW;
+ }
+
+ if (terminalAcknowledgementAssigned) {
+ registry.acknowledgementsFlushed(recordSupplier.flushAcknowledgementsSync());
+ }
+ }
+
+ void requestStop()
+ {
+ stopRequested.set(true);
+ completions.offer(ShareGroupStagingCompletion.stop());
Review Comment:
## CodeQL / Ignored error status of call
Method requestStop ignores exceptional return value of BlockingQueue<ShareGroupStagingCompletion>.offer.
[Show more details](https://github.com/apache/druid/security/code-scanning/12024)
##########
indexing-service/src/test/java/org/apache/druid/indexing/common/actions/SegmentTransactionalAppendFromShareInboxActionTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import org.apache.druid.error.DruidException;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.task.NoopTask;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.GlobalTaskLockbox;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.LockResult;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.overlord.ShareInboxBatch;
+import org.apache.druid.indexing.overlord.ShareInboxClaimRequest;
+import org.apache.druid.indexing.overlord.ShareInboxCompletionRequest;
+import org.apache.druid.indexing.overlord.ShareInboxManifest;
+import org.apache.druid.indexing.overlord.TimeChunkLockRequest;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.metrics.StubServiceEmitter;
+import org.apache.druid.segment.TestHelper;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.joda.time.Interval;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class SegmentTransactionalAppendFromShareInboxActionTest
+{
+ private static final String DATA_SOURCE = "datasource";
+ private static final String TASK_ID = "task-1";
+ private static final Interval INTERVAL = Intervals.of("2026-01-01/2026-01-02");
+
+ @RegisterExtension
+ public final TaskActionTestKit actionTestKit = new TaskActionTestKit();
+
+ @Test
+ public void testSerde() throws Exception
+ {
+ final SegmentTransactionalAppendFromShareInboxAction original = new SegmentTransactionalAppendFromShareInboxAction(
+ Set.of(),
+ completionRequest("manifest-1", 2L),
+ null
+ );
+
+ final TaskAction<?> deserialized = TestHelper.JSON_MAPPER.readValue(
+ TestHelper.JSON_MAPPER.writeValueAsBytes(original),
+ TaskAction.class
+ );
+
+ Assertions.assertInstanceOf(SegmentTransactionalAppendFromShareInboxAction.class, deserialized);
+ final SegmentTransactionalAppendFromShareInboxAction action =
+ (SegmentTransactionalAppendFromShareInboxAction) deserialized;
+ Assertions.assertEquals(original.getSegments(), action.getSegments());
+ Assertions.assertEquals(original.getCompletionRequest().getClaims(), action.getCompletionRequest().getClaims());
+ }
+
+ @Test
+ public void testPerformCompletesZeroRowManifest()
+ {
+ final NoopTask task = task();
+ final ShareInboxCompletionRequest request = completionRequest("manifest-1", 2L);
+ final SegmentTransactionalAppendFromShareInboxAction action =
+ new SegmentTransactionalAppendFromShareInboxAction(Set.of(), request, null);
+ final TaskActionToolbox toolbox = mock(TaskActionToolbox.class);
+ final GlobalTaskLockbox lockbox = mock(GlobalTaskLockbox.class);
+ final IndexerMetadataStorageCoordinator coordinator = mock(IndexerMetadataStorageCoordinator.class);
+ final SegmentPublishResult expected = SegmentPublishResult.ok(Set.of());
+ when(toolbox.getTaskLockbox()).thenReturn(lockbox);
+ when(toolbox.getIndexerMetadataStorageCoordinator()).thenReturn(coordinator);
+ when(toolbox.getEmitter()).thenReturn(new StubServiceEmitter());
+ when(lockbox.findLocksForTask(task)).thenReturn(List.of());
+ when(coordinator.commitAppendSegmentsAndShareInbox(Set.of(), Map.of(), TASK_ID, null, request))
+ .thenReturn(expected);
+
+ Assertions.assertSame(expected, action.perform(task, toolbox));
+ verify(coordinator).commitAppendSegmentsAndShareInbox(Set.of(), Map.of(), TASK_ID, null, request);
+ }
+
+ @Test
+ public void testPerformRejectsDifferentClaimOwner()
+ {
+ final NoopTask task = task();
+ final ShareInboxCompletionRequest request = new ShareInboxCompletionRequest(
+ DATA_SOURCE,
+ "inbox",
+ "fingerprint",
+ "other-task",
+ Map.of("manifest-1", 2L),
+ "completion-1"
+ );
+ final SegmentTransactionalAppendFromShareInboxAction action =
+ new SegmentTransactionalAppendFromShareInboxAction(Set.of(), request, null);
+ final TaskActionToolbox toolbox = mock(TaskActionToolbox.class);
+ final IndexerMetadataStorageCoordinator coordinator = mock(IndexerMetadataStorageCoordinator.class);
+ when(toolbox.getIndexerMetadataStorageCoordinator()).thenReturn(coordinator);
+
+ Assertions.assertThrows(DruidException.class, () -> action.perform(task, toolbox));
+ verify(coordinator, never()).commitAppendSegmentsAndShareInbox(any(), any(), any(), any(), any());
+ }
+
+ @Test
+ public void testPerformRejectsTaskWithoutPendingSegmentAllocation()
+ {
+ final Task task = mock(Task.class);
+ final SegmentTransactionalAppendFromShareInboxAction action =
+ new SegmentTransactionalAppendFromShareInboxAction(Set.of(), completionRequest("manifest-1", 2L), null);
+ when(task.getId()).thenReturn(TASK_ID);
+ when(task.getType()).thenReturn("test");
+
+ Assertions.assertThrows(DruidException.class, () -> action.perform(task, mock(TaskActionToolbox.class)));
+ }
+
+ @Test
+ public void testPerformPublishesSegmentAndCompletesInbox() throws Exception
+ {
+ actionTestKit.getTestDerbyConnector().createShareReceiptsTable();
+ actionTestKit.getTestDerbyConnector().createShareInboxTable();
+ final NoopTask task = task();
+ actionTestKit.getTaskLockbox().add(task);
+ final LockResult lockResult = actionTestKit.getTaskLockbox().lock(
+ task,
+ new TimeChunkLockRequest(TaskLockType.APPEND, task, INTERVAL, null),
+ 5_000
+ );
+ Assertions.assertTrue(lockResult.isOk());
+ final DataSegment segment = segment(lockResult.getTaskLock().getVersion());
+ final IndexerMetadataStorageCoordinator coordinator = actionTestKit.getMetadataStorageCoordinator();
+ coordinator.stageShareInboxBatch(batch());
+ final ShareInboxManifest manifest = coordinator.claimShareInboxManifests(
+ new ShareInboxClaimRequest(DATA_SOURCE, "inbox", "fingerprint", TASK_ID, 1, 10, 10_000, 60_000)
+ ).getManifests().get(0);
+ final ShareInboxCompletionRequest request = completionRequest(manifest.getManifestId(), manifest.getClaimEpoch());
+ final SegmentTransactionalAppendFromShareInboxAction action =
+ new SegmentTransactionalAppendFromShareInboxAction(Set.of(segment), request, null);
+
+ final SegmentPublishResult first = action.perform(task, actionTestKit.getTaskActionToolbox());
+ final SegmentPublishResult retry = action.perform(task, actionTestKit.getTaskActionToolbox());
+
+ Assertions.assertTrue(first.isSuccess());
+ Assertions.assertEquals(Set.of(segment), first.getSegments());
+ Assertions.assertTrue(retry.isSuccess());
+ Assertions.assertTrue(retry.getSegments().isEmpty());
+ Assertions.assertEquals(segment, coordinator.retrieveSegmentForId(segment.getId()));
+ }
+
+ private static NoopTask task()
+ {
+ return new NoopTask(TASK_ID, null, DATA_SOURCE, 0, 0, null);
+ }
+
+ private static ShareInboxCompletionRequest completionRequest(String manifestId, long claimEpoch)
+ {
+ return new ShareInboxCompletionRequest(
+ DATA_SOURCE,
+ "inbox",
+ "fingerprint",
+ TASK_ID,
+ Map.of(manifestId, claimEpoch),
+ "completion-1"
+ );
+ }
+
+ private static ShareInboxBatch batch()
+ {
+ return new ShareInboxBatch(
+ DATA_SOURCE,
+ "inbox",
+ "group",
+ "cluster",
+ "topic-id",
+ "topic",
+ 0,
+ "fingerprint",
+ "manifest-1",
+ "path/manifest-1",
+ "hash",
+ 1,
+ List.of(1L),
+ 100
+ );
+ }
+
+ private static DataSegment segment(String version)
+ {
+ return new DataSegment(
+ DATA_SOURCE,
+ INTERVAL,
+ version,
+ Map.of(),
+ List.of("dim"),
+ List.of("metric"),
+ new LinearShardSpec(0),
+ 9,
+ 100
+ );
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [DataSegment.DataSegment](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/12030)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/ShareGroupAcquisitionLoopTest.java:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+
+public class ShareGroupAcquisitionLoopTest
+{
+ @Test
+ public void testRepeatedRenewalsStartOneStagingOperation() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.empty());
+ for (int i = 0; i < 11; i++) {
+ supplier.addPoll(record("topic", 0, 1));
+ }
+ final RecordingExecutor executor = new RecordingExecutor();
+ final AtomicInteger stageCount = new AtomicInteger();
+ final ShareGroupAcquisitionLoop loop = loop(
+ supplier,
+ records -> stageCount.incrementAndGet(),
+ executor,
+ registry(10, 1_000)
+ );
+
+ for (int i = 0; i < 10; i++) {
+ loop.runOnce();
+ }
+ Assertions.assertEquals(1, executor.size());
+ Assertions.assertEquals(0, stageCount.get());
+ Assertions.assertEquals(10, supplier.acknowledgements.size());
+ Assertions.assertTrue(
+ supplier.acknowledgements.stream().allMatch(ack -> ack.type == AcknowledgeType.RENEW)
+ );
+
+ executor.runNext();
+ loop.runOnce();
+
+ Assertions.assertEquals(1, stageCount.get());
+ Assertions.assertEquals(AcknowledgeType.ACCEPT, supplier.acknowledgements.get(10).type);
+ Assertions.assertEquals(1, supplier.flushCount);
+ }
+
+ @Test
+ public void testSaturatedRecordIsReleasedWithoutSecondSubmission() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.empty());
+ supplier.addPoll(record("topic", 0, 1), record("topic", 0, 2));
+ final RecordingExecutor executor = new RecordingExecutor();
+ final ShareGroupAcquisitionLoop loop = loop(
+ supplier,
+ records -> {},
+ executor,
+ registry(1, 1_000)
+ );
+
+ loop.runOnce();
+
+ Assertions.assertEquals(1, executor.size());
+ Assertions.assertEquals(AcknowledgeType.RELEASE, supplier.acknowledgement(2).type);
+ Assertions.assertEquals(AcknowledgeType.RENEW, supplier.acknowledgement(1).type);
+ Assertions.assertEquals(1, supplier.flushCount);
+ }
+
+ @Test
+ public void testByteSaturationReleasesRecord() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.empty());
+ supplier.addPoll(record("topic", 0, 1), record("topic", 0, 2));
+ final ShareGroupAcquisitionRegistry registry = registry(2, 1);
+ final RecordingExecutor executor = new RecordingExecutor();
+ final ShareGroupAcquisitionLoop loop = new ShareGroupAcquisitionLoop(
+ "topic",
+ supplier,
+ records -> {},
+ executor,
+ registry,
+ ignored -> 1,
+ 1,
+ 60_000
+ );
+
+ loop.runOnce();
+
+ Assertions.assertEquals(1, executor.size());
+ Assertions.assertEquals(AcknowledgeType.RELEASE, supplier.acknowledgement(2).type);
+ }
+
+ @Test
+ public void testExecutorRejectionReleasesRecord() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.of(60_000));
+ supplier.addPoll(record("topic", 0, 1));
+ final Executor rejectingExecutor = command -> {
+ throw new RejectedExecutionException("full");
+ };
+ final ShareGroupAcquisitionRegistry registry = registry(1, 1_000);
+ final ShareGroupAcquisitionLoop loop = loop(supplier, records -> {}, rejectingExecutor, registry);
+
+ loop.runOnce();
+
+ Assertions.assertEquals(AcknowledgeType.RELEASE, supplier.acknowledgement(1).type);
+ Assertions.assertEquals(1, supplier.flushCount);
+ Assertions.assertEquals(0, registry.size());
+ }
+
+ @Test
+ public void testWorkerSuccessIsAcceptedOnOwnerThread() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.of(60_000));
+ supplier.addPoll(record("topic", 0, 1));
+ final AtomicReference<Thread> stagingThread = new AtomicReference<>();
+ final ExecutorService stagingExecutor = Executors.newSingleThreadExecutor();
+ try {
+ final ShareGroupAcquisitionLoop loop = loop(
+ supplier,
+ records -> stagingThread.set(Thread.currentThread()),
+ stagingExecutor,
+ registry(1, 1_000)
+ );
+
+ loop.runOnce();
+
+ Assertions.assertEquals(AcknowledgeType.ACCEPT, supplier.acknowledgement(1).type);
+ Assertions.assertNotSame(stagingThread.get(), supplier.acknowledgement(1).thread);
+ Assertions.assertEquals(Set.of(Thread.currentThread()), supplier.consumerThreads);
+ }
+ finally {
+ stagingExecutor.shutdownNow();
+ }
+ }
+
+ @Test
+ public void testWorkerFailureIsReleasedOnOwnerThread() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.of(60_000));
+ supplier.addPoll(record("topic", 0, 1));
+ final ExecutorService stagingExecutor = Executors.newSingleThreadExecutor();
+ try {
+ final ShareGroupAcquisitionLoop loop = loop(
+ supplier,
+ records -> {
+ throw new IllegalStateException("upload failed");
+ },
+ stagingExecutor,
+ registry(1, 1_000)
+ );
+
+ loop.runOnce();
+
+ Assertions.assertEquals(AcknowledgeType.RELEASE, supplier.acknowledgement(1).type);
+ Assertions.assertEquals(Set.of(Thread.currentThread()), supplier.consumerThreads);
+ }
+ finally {
+ stagingExecutor.shutdownNow();
+ }
+ }
+
+ @Test
+ public void testStopDuringUploadReleasesAndFlushes() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.of(60_000));
+ supplier.addPoll(record("topic", 0, 1));
+ final RecordingExecutor stagingExecutor = new RecordingExecutor();
+ final ShareGroupAcquisitionLoop loop = loop(
+ supplier,
+ records -> {},
+ stagingExecutor,
+ registry(1, 1_000)
+ );
+ final ExecutorService ownerExecutor = Executors.newSingleThreadExecutor();
+ try {
+ final Future<?> run = ownerExecutor.submit(() -> {
+ try {
+ loop.runOnce();
+ }
+ catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ Assertions.assertTrue(supplier.pollReturned.await(10, TimeUnit.SECONDS));
+
+ loop.requestStop();
+ run.get(10, TimeUnit.SECONDS);
+
+ Assertions.assertEquals(AcknowledgeType.RELEASE, supplier.acknowledgement(1).type);
+ Assertions.assertEquals(1, supplier.flushCount);
+ Assertions.assertEquals(1, supplier.wakeupCount);
+ }
+ finally {
+ ownerExecutor.shutdownNow();
+ }
+ }
+
+ @Test
+ public void testFailedFlushRetainsTerminalEntry() throws Exception
+ {
+ final ScriptedSupplier supplier = new ScriptedSupplier(Optional.of(60_000));
+ supplier.flushFailure = new IllegalStateException("flush failed");
+ supplier.addPoll(record("topic", 0, 1));
+ final ShareGroupAcquisitionRegistry registry = registry(1, 1_000);
+ final ShareGroupAcquisitionLoop loop = loop(supplier, records -> {}, Runnable::run, registry);
+
+ loop.runOnce();
+
+ Assertions.assertEquals(1, registry.size());
+ Assertions.assertEquals(
+ ShareGroupAcquisitionRegistry.State.ACCEPT_PENDING,
+ registry.get(new ShareGroupRecordIdentity("topic", 0, 1)).orElseThrow().getState()
+ );
+ }
+
+ private static ShareGroupAcquisitionLoop loop(
+ ScriptedSupplier supplier,
+ ShareGroupBatchStager stager,
+ Executor executor,
+ ShareGroupAcquisitionRegistry registry
+ )
+ {
+ return new ShareGroupAcquisitionLoop(
+ "topic",
+ supplier,
+ stager,
+ executor,
+ registry,
+ ignored -> 1,
+ 1,
+ 60_000
+ );
+ }
+
+ private static ShareGroupAcquisitionRegistry registry(int maxRecords, long maxBytes)
+ {
+ final AtomicLong nanoTime = new AtomicLong();
+ return new ShareGroupAcquisitionRegistry(maxRecords, maxBytes, 0.5, nanoTime::get);
+ }
+
+ @SafeVarargs
+ private static List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records(
+ OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>... records
+ )
+ {
+ return List.of(records);
+ }
+
+ private static OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record(
+ String topic,
+ int partition,
+ long offset
+ )
+ {
+ final ConsumerRecord<byte[], byte[]> consumerRecord = new ConsumerRecord<>(
+ topic,
+ partition,
+ offset,
+ null,
+ new byte[]{1}
+ );
+ return new OrderedPartitionableRecord<>(
+ topic,
+ new KafkaTopicPartition(true, topic, partition),
+ offset,
+ List.of(new KafkaRecordEntity(consumerRecord))
+ );
+ }
+
+ private static final class RecordingExecutor implements Executor
+ {
+ private final Deque<Runnable> commands = new ArrayDeque<>();
+
+ @Override
+ public void execute(Runnable command)
+ {
+ commands.add(command);
+ }
+
+ int size()
+ {
+ return commands.size();
+ }
+
+ void runNext()
+ {
+ commands.remove().run();
+ }
+ }
+
+ private static final class RecordedAcknowledgement
+ {
+ private final long offset;
+ private final AcknowledgeType type;
+ private final Thread thread;
+
+ private RecordedAcknowledgement(long offset, AcknowledgeType type, Thread thread)
+ {
+ this.offset = offset;
+ this.type = type;
+ this.thread = thread;
+ }
+ }
+
+ private static final class ScriptedSupplier
+ implements AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>
+ {
+ private final Optional<Integer> lockTimeoutMs;
+ private final Deque<List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>>> polls =
+ new ArrayDeque<>();
+ private final List<RecordedAcknowledgement> acknowledgements = new ArrayList<>();
+ private final Set<Thread> consumerThreads = new HashSet<>();
+ private final CountDownLatch pollReturned = new CountDownLatch(1);
+ private Set<String> subscription = Set.of();
+ private int flushCount;
+ private int wakeupCount;
+ private Exception flushFailure;
+
+ private ScriptedSupplier(Optional<Integer> lockTimeoutMs)
+ {
+ this.lockTimeoutMs = lockTimeoutMs;
+ }
+
+ @SafeVarargs
+ final void addPoll(OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>... pollRecords)
+ {
+ polls.add(records(pollRecords));
+ }
+
+ RecordedAcknowledgement acknowledgement(long offset)
+ {
+ return acknowledgements.stream()
+ .filter(acknowledgement -> acknowledgement.offset == offset)
+ .findFirst()
+ .orElseThrow();
+ }
+
+ @Override
+ public void subscribe(Set<String> topics)
+ {
+ consumerThreads.add(Thread.currentThread());
+ subscription = Set.copyOf(topics);
+ }
+
+ @Override
+ public void unsubscribe()
+ {
+ consumerThreads.add(Thread.currentThread());
+ subscription = Set.of();
+ }
+
+ @Override
+ public Set<String> subscription()
+ {
+ consumerThreads.add(Thread.currentThread());
+ return subscription;
+ }
+
+ @Override
+ public List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> poll(long timeoutMs)
+ {
+ consumerThreads.add(Thread.currentThread());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ polls.isEmpty() ? List.of() : polls.remove();
+ pollReturned.countDown();
+ return records;
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset)
+ {
+ acknowledge(partitionId, offset, AcknowledgeType.ACCEPT);
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset, AcknowledgeType type)
+ {
+ consumerThreads.add(Thread.currentThread());
+ acknowledgements.add(new RecordedAcknowledgement(offset, type, Thread.currentThread()));
+ }
+
+ @Override
+ public void acknowledge(
+ Map<KafkaTopicPartition, Collection<Long>> offsets,
+ AcknowledgeType type
+ )
+ {
+ for (Map.Entry<KafkaTopicPartition, Collection<Long>> entry : offsets.entrySet()) {
+ for (Long offset : entry.getValue()) {
+ acknowledge(entry.getKey(), offset, type);
+ }
+ }
+ }
+
+ @Override
+ public Map<KafkaTopicPartition, Optional<Exception>> flushAcknowledgementsSync()
+ {
+ consumerThreads.add(Thread.currentThread());
+ flushCount++;
+ final Map<KafkaTopicPartition, Optional<Exception>> result = new HashMap<>();
+ for (RecordedAcknowledgement acknowledgement : acknowledgements) {
Review Comment:
## CodeQL / Unread local variable
Variable 'RecordedAcknowledgement acknowledgement' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/12023)
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplierTest.java:
##########
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class KafkaShareGroupRecordSupplierTest
+{
+ private KafkaShareConsumer<byte[], byte[]> mockConsumer;
+ private KafkaShareGroupRecordSupplier supplier;
+
+ @SuppressWarnings("unchecked")
+ @BeforeEach
+ public void setUp()
+ {
+ mockConsumer = mock(KafkaShareConsumer.class);
+ supplier = new KafkaShareGroupRecordSupplier(mockConsumer);
+ }
+
+ @AfterEach
+ public void tearDown()
+ {
+ supplier.close();
+ }
+
+ @Test
+ public void testSubscribeAndSubscription()
+ {
+ final Set<String> topics = Set.of("topic-a", "topic-b");
+ when(mockConsumer.subscription()).thenReturn(topics);
+
+ supplier.subscribe(topics);
+ verify(mockConsumer).subscribe(topics);
+
+ Assertions.assertEquals(topics, supplier.subscription());
+ }
+
+ @Test
+ public void testUnsubscribe()
+ {
+ supplier.unsubscribe();
+ verify(mockConsumer).unsubscribe();
+ }
+
+ @Test
+ public void testPollWrapsRecords()
+ {
+ final ConsumerRecord<byte[], byte[]> record1 = new ConsumerRecord<>(
+ "test-topic", 0, 100L, "key1".getBytes(StandardCharsets.UTF_8), "value1".getBytes(StandardCharsets.UTF_8)
+ );
+ final ConsumerRecord<byte[], byte[]> record2 = new ConsumerRecord<>(
+ "test-topic", 1, 200L, "key2".getBytes(StandardCharsets.UTF_8), "value2".getBytes(StandardCharsets.UTF_8)
+ );
+
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record1));
+ recordMap.put(new TopicPartition("test-topic", 1), List.of(record2));
+ final ConsumerRecords<byte[], byte[]> consumerRecords = new ConsumerRecords<>(recordMap);
+
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(consumerRecords);
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(2, result.size());
+
+ // Verify first record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled1 =
+ result.stream().filter(r -> r.getSequenceNumber() == 100L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled1);
+ Assertions.assertEquals("test-topic", polled1.getStream());
+ Assertions.assertEquals(0, polled1.getPartitionId().partition());
+ Assertions.assertNotNull(polled1.getData());
+ Assertions.assertEquals(1, polled1.getData().size());
+
+ // Verify second record
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> polled2 =
+ result.stream().filter(r -> r.getSequenceNumber() == 200L).findFirst().orElse(null);
+ Assertions.assertNotNull(polled2);
+ Assertions.assertEquals(1, polled2.getPartitionId().partition());
+ }
+
+ @Test
+ public void testPollWithNullValue()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 50L, "key".getBytes(StandardCharsets.UTF_8), null
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), List.of(record));
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(new ConsumerRecords<>(recordMap));
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(1000);
+
+ Assertions.assertEquals(1, result.size());
+ Assertions.assertTrue(result.get(0).getData().isEmpty());
+ Assertions.assertSame(record, ((KafkaShareGroupRecord) result.get(0)).getConsumerRecord());
+ }
+
+ @Test
+ public void testPollReturnsEmptyOnTimeout()
+ {
+ when(mockConsumer.poll(any(Duration.class))).thenReturn(ConsumerRecords.empty());
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> result =
+ supplier.poll(100);
+ Assertions.assertTrue(result.isEmpty());
+ }
+
+ @Test
+ public void testAcknowledgeDefaultAccept()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 42L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 42L);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRelease()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.RELEASE);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithReject()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 10L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 10L, AcknowledgeType.REJECT);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.REJECT)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeWithRenew()
+ {
+ final ConsumerRecord<byte[], byte[]> record = new ConsumerRecord<>(
+ "test-topic", 0, 99L, "key".getBytes(StandardCharsets.UTF_8), "value".getBytes(StandardCharsets.UTF_8)
+ );
+ pollSingleRecord(record);
+
+ final KafkaTopicPartition partition = new KafkaTopicPartition(true, "test-topic", 0);
+ supplier.acknowledge(partition, 99L, AcknowledgeType.RENEW);
+
+ Mockito.verify(mockConsumer).acknowledge(
+ Mockito.same(record),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.RENEW)
+ );
+ }
+
+ @Test
+ public void testAcknowledgeBatch()
+ {
+ final List<ConsumerRecord<byte[], byte[]>> recordsP0 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 0, 1L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 2L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 0, 3L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final List<ConsumerRecord<byte[], byte[]>> recordsP1 = Arrays.asList(
+ new ConsumerRecord<>("test-topic", 1, 10L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8)),
+ new ConsumerRecord<>("test-topic", 1, 11L, "k".getBytes(StandardCharsets.UTF_8), "v".getBytes(StandardCharsets.UTF_8))
+ );
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition("test-topic", 0), recordsP0);
+ recordMap.put(new TopicPartition("test-topic", 1), recordsP1);
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
+ supplier.poll(1000);
+
+ final KafkaTopicPartition p0 = new KafkaTopicPartition(true, "test-topic", 0);
+ final KafkaTopicPartition p1 = new KafkaTopicPartition(true, "test-topic", 1);
+
+ final Map<KafkaTopicPartition, java.util.Collection<Long>> offsets = new HashMap<>();
+ offsets.put(p0, Arrays.asList(1L, 2L, 3L));
+ offsets.put(p1, Arrays.asList(10L, 11L));
+
+ supplier.acknowledge(offsets, AcknowledgeType.ACCEPT);
+
+ Mockito.verify(mockConsumer, Mockito.times(5)).acknowledge(
+ Mockito.<ConsumerRecord<byte[], byte[]>>any(),
+ Mockito.eq(org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT)
+ );
+ }
+
+ private void pollSingleRecord(ConsumerRecord<byte[], byte[]> record)
+ {
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> recordMap = new HashMap<>();
+ recordMap.put(new TopicPartition(record.topic(), record.partition()), List.of(record));
+ Mockito.when(mockConsumer.poll(Mockito.any(Duration.class)))
+ .thenReturn(new ConsumerRecords<>(recordMap));
Review Comment:
## CodeQL / Deprecated method or constructor invocation
Invoking [ConsumerRecords.ConsumerRecords](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/12029)
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/RenewShareInboxClaimsAction.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.ShareInboxRenewRequest;
+import org.apache.druid.indexing.overlord.ShareInboxRenewResult;
+
+import java.util.Objects;
+
+public class RenewShareInboxClaimsAction implements TaskAction<ShareInboxRenewResult>
+{
+ public static final String TYPE = "renewShareInboxClaims";
+
+ private final ShareInboxRenewRequest request;
+
+ @JsonCreator
+ public RenewShareInboxClaimsAction(@JsonProperty("request") ShareInboxRenewRequest request)
+ {
+ this.request = Objects.requireNonNull(request, "request");
+ }
+
+ @JsonProperty
+ public ShareInboxRenewRequest getRequest()
+ {
+ return request;
+ }
+
+ @Override
+ public TypeReference<ShareInboxRenewResult> getReturnTypeReference()
+ {
+ return new TypeReference<>() {};
+ }
+
+ @Override
+ public ShareInboxRenewResult perform(Task task, TaskActionToolbox toolbox)
+ {
+ ClaimShareInboxManifestsAction.validateTask(task, request.getDataSource(), request.getClaimOwner());
+ return toolbox.getIndexerMetadataStorageCoordinator().renewShareInboxClaims(request);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "RenewShareInboxClaimsAction{" +
+ "request=" + request +
Review Comment:
## CodeQL / Use of default toString()
Default toString(): ShareInboxRenewRequest inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/druid/security/code-scanning/12026)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3202697283
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Sure, I am working on it and need full round of local validation. I will share the code changes soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3259153370
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ return runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ }
+ finally {
+ activeSupplier.set(null);
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
Review Comment:
[P2] Close the appenderator when setup fails
After this refactor, appenderator.closeNow() only runs in the try/finally that starts after recordSupplier.subscribe(...) and driver.startJob(...). If supplier creation, subscribe, or startJob throws, run() only calls driver.close(), and BaseAppenderatorDriver.close explicitly does not close the underlying Appenderator, so failed startup can leak realtime appenderator resources. Keep the appenderator close guard in run() or include setup in runLoop's try/finally.
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Throwables;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/** Verifies that ShareGroupIndexTask can be submitted via raw JSON through the Overlord HTTP API. */
+public class ShareGroupIndexTaskJsonSubmitIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_jsonSubmit_csvFormat_ingestsData() throws Exception
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+ kafkaServer.setShareGroupAutoOffsetReset("json-submit-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "json-submit-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 8;
+ kafkaServer.publishRecordsToTopic(topic, generateRecords(numRecords));
+
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(numRecords)
+ );
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ final long deadlineMs = System.currentTimeMillis() + 30_000L;
+ while (System.currentTimeMillis() < deadlineMs) {
+ try {
+ cluster.callApi().verifySqlQuery("SELECT COUNT(*) FROM %s", dataSource, String.valueOf(numRecords));
+ break;
+ }
+ catch (AssertionError ignored) {
+ Thread.sleep(500);
+ }
+ }
+
+ cluster.callApi().onLeaderOverlord(o -> o.cancelTask(task.getId()));
+ cluster.callApi().waitForTaskToFinish(task.getId(), overlord.latchableEmitter());
+ }
+
+ @Test
+ public void test_jsonSubmit_roundTripsPayload() throws Exception
+ {
+ final String topic = dataSource + "_roundtrip_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset("roundtrip-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "roundtrip-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final org.apache.druid.client.indexing.TaskPayloadResponse payloadResponse =
+ cluster.callApi().onLeaderOverlord(o -> o.taskPayload(task.getId()));
+ Assertions.assertEquals(task.getId(), payloadResponse.getTask());
+
+ final JsonNode payloadJson = mapper.valueToTree(payloadResponse.getPayload());
+ Assertions.assertEquals("share_group_index", payloadJson.path("type").asText());
Review Comment:
[P2] Fix JSON-submit payload expectations
This new IT serializes a real ShareGroupIndexTask and reads the Overlord payload, but the task is registered as index_kafka_share_group and serializes ioConfig at the top level, not under spec. The round-trip test therefore fails on the type assertion, and still fails on the topic/group assertions after that is fixed; the malformed-json test below also uses the wrong type/shape, so it does not exercise the intended missing-field path.
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexTaskJsonSubmitIT.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Throwables;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/** Verifies that ShareGroupIndexTask can be submitted via raw JSON through the Overlord HTTP API. */
+public class ShareGroupIndexTaskJsonSubmitIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(30)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_jsonSubmit_csvFormat_ingestsData() throws Exception
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+ kafkaServer.setShareGroupAutoOffsetReset("json-submit-group", "earliest");
+
+ final ShareGroupIndexTask task = buildTask(topic, "json-submit-group");
+ final JsonNode taskJson = mapper.readTree(mapper.writeValueAsString(task));
+ cluster.callApi().onLeaderOverlord(o -> o.runTask(task.getId(), taskJson));
+
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ final int numRecords = 8;
+ kafkaServer.publishRecordsToTopic(topic, generateRecords(numRecords));
+
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(numRecords)
+ );
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ final long deadlineMs = System.currentTimeMillis() + 30_000L;
Review Comment:
[P3] Fail when the row-count wait expires
If verifySqlQuery never observes the expected count, this loop exits when the deadline expires and the test continues to cancel the task without failing. A broken JSON-submit ingestion/query path can pass after 30 seconds. Track the last assertion and fail after the deadline, or reuse the existing assertRowCountEventually pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3241286739
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaShareGroupRecordSupplier.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.KafkaShareConsumer;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+
+import javax.annotation.Nonnull;
+import javax.validation.constraints.NotNull;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Adapts {@link KafkaShareConsumer} to {@link AcknowledgingRecordSupplier}.
+ * Delivery state lives on the broker; the supplier sets {@code group.id} to
+ * the configured share group name.
+ */
+public class KafkaShareGroupRecordSupplier
+ implements AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>
+{
+ private static final Logger log = new Logger(KafkaShareGroupRecordSupplier.class);
+
+ private final KafkaShareConsumer<byte[], byte[]> consumer;
+
+ /**
+ * Records returned by the most recent {@link #poll(long)}, retained so
+ * {@link #acknowledge} can pass the original {@link ConsumerRecord} to the
+ * Kafka client. The {@code (topic, partition, offset)} ack variant is fragile
+ * once the share-fetch buffer rolls over (KIP-932).
+ */
+ private final Map<RecordKey, ConsumerRecord<byte[], byte[]>> deliveredRecords = new HashMap<>();
+ private boolean closed;
+
+ public KafkaShareGroupRecordSupplier(
+ Map<String, Object> consumerProperties,
+ ObjectMapper sortingMapper,
+ String groupId
+ )
+ {
+ this(createShareConsumer(consumerProperties, sortingMapper, groupId));
+ }
+
+ @VisibleForTesting
+ public KafkaShareGroupRecordSupplier(KafkaShareConsumer<byte[], byte[]> consumer)
+ {
+ this.consumer = consumer;
+ }
+
+ @Override
+ public void subscribe(Set<String> topics)
+ {
+ consumer.subscribe(topics);
+ }
+
+ @Override
+ public void unsubscribe()
+ {
+ consumer.unsubscribe();
+ }
+
+ @Override
+ public Set<String> subscription()
+ {
+ return consumer.subscription();
+ }
+
+ @NotNull
+ @Override
+ public List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> poll(long timeoutMs)
+ {
+ deliveredRecords.clear();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> polledRecords =
+ new ArrayList<>();
+ for (ConsumerRecord<byte[], byte[]> record : consumer.poll(Duration.ofMillis(timeoutMs))) {
+ deliveredRecords.put(new RecordKey(record.topic(), record.partition(), record.offset()), record);
+ polledRecords.add(new OrderedPartitionableRecord<>(
+ record.topic(),
+ new KafkaTopicPartition(true, record.topic(), record.partition()),
+ record.offset(),
+ record.value() == null ? null : ImmutableList.of(new KafkaRecordEntity(record)),
+ record.timestamp()
+ ));
+ }
+ return polledRecords;
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset)
+ {
+ acknowledge(partitionId, offset, AcknowledgeType.ACCEPT);
+ }
+
+ @Override
+ public void acknowledge(KafkaTopicPartition partitionId, Long offset, AcknowledgeType type)
+ {
+ final String topic = partitionId.topic().orElseThrow(
+ () -> new IllegalArgumentException("Cannot acknowledge record without topic")
+ );
+ final ConsumerRecord<byte[], byte[]> record = deliveredRecords.get(
+ new RecordKey(topic, partitionId.partition(), offset)
+ );
+ if (record == null) {
+ throw new IllegalStateException(StringUtils.format(
+ "Cannot acknowledge unknown record at topic[%s] partition[%d] offset[%d]; "
+ + "either it was not delivered by the most recent poll() or it has already been acknowledged.",
+ topic, partitionId.partition(), offset
+ ));
+ }
+ consumer.acknowledge(record, toKafkaAcknowledgeType(type));
+ }
+
+ @Override
+ public void acknowledge(
+ Map<KafkaTopicPartition, Collection<Long>> offsets,
+ AcknowledgeType type
+ )
+ {
+ final org.apache.kafka.clients.consumer.AcknowledgeType kafkaType = toKafkaAcknowledgeType(type);
+ for (Map.Entry<KafkaTopicPartition, Collection<Long>> entry : offsets.entrySet()) {
+ final KafkaTopicPartition partition = entry.getKey();
+ final String topic = partition.topic().orElseThrow(
+ () -> new IllegalArgumentException("Cannot acknowledge record without topic")
+ );
+ for (Long offset : entry.getValue()) {
+ final ConsumerRecord<byte[], byte[]> record = deliveredRecords.get(
+ new RecordKey(topic, partition.partition(), offset)
+ );
+ if (record == null) {
+ throw new IllegalStateException(StringUtils.format(
+ "Cannot acknowledge unknown record at topic[%s] partition[%d] offset[%d]; "
+ + "either it was not delivered by the most recent poll() or it has already been acknowledged.",
+ topic, partition.partition(), offset
+ ));
+ }
+ consumer.acknowledge(record, kafkaType);
+ }
+ }
+ }
+
+ @Override
+ public Map<KafkaTopicPartition, Optional<Exception>> commitSync()
+ {
+ final Map<TopicIdPartition, Optional<KafkaException>> result = consumer.commitSync();
+ final Map<KafkaTopicPartition, Optional<Exception>> mapped = new HashMap<>();
+ for (Map.Entry<TopicIdPartition, Optional<KafkaException>> entry : result.entrySet()) {
+ final TopicIdPartition tip = entry.getKey();
+ mapped.put(
+ new KafkaTopicPartition(true, tip.topic(), tip.partition()),
+ entry.getValue().map(e -> (Exception) e)
+ );
+ }
+ return mapped;
+ }
+
+ @Override
+ public Set<KafkaTopicPartition> getPartitionIds(String stream)
+ {
+ // Share consumer does not expose partition assignment; broker manages it.
+ return Collections.emptySet();
+ }
+
+ @Override
+ public void wakeup()
+ {
+ consumer.wakeup();
+ }
+
+ @Override
+ public Optional<Integer> acquisitionLockTimeoutMs()
+ {
+ return consumer.acquisitionLockTimeoutMs();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ try {
+ consumer.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing KafkaShareConsumer");
+ }
+ }
+
+ @Nonnull
+ private static org.apache.kafka.clients.consumer.AcknowledgeType toKafkaAcknowledgeType(AcknowledgeType type)
+ {
+ switch (type) {
+ case ACCEPT:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.ACCEPT;
+ case RELEASE:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.RELEASE;
+ case REJECT:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.REJECT;
+ case RENEW:
+ return org.apache.kafka.clients.consumer.AcknowledgeType.RENEW;
+ default:
+ throw new IllegalArgumentException("Unknown acknowledge type: " + type);
+ }
+ }
+
+ private static KafkaShareConsumer<byte[], byte[]> createShareConsumer(
+ Map<String, Object> consumerProperties,
+ ObjectMapper sortingMapper,
+ String groupId
+ )
+ {
+ final Map<String, Object> sanitized = ShareGroupConsumerProperties.sanitize(consumerProperties);
Review Comment:
[P2] Unsupported share-consumer configs can still enter through dynamic config
sanitize() runs before KafkaRecordSupplier.addConsumerPropertiesFromConfig, but that helper expands druid.dynamic.config.provider afterward and copies all returned entries directly into the Properties. If a dynamic provider supplies a forbidden key such as enable.auto.commit, auto.offset.reset, or group.instance.id, KafkaShareConsumer still receives it and the task can fail at startup despite the documented stripping behavior. Sanitize after dynamic expansion as well, or expand into a map first and then strip unsupported keys once.
##########
docs/ingestion/kafka-share-group-ingestion.md:
##########
@@ -0,0 +1,337 @@
+---
+id: kafka-share-group-ingestion
+title: "Kafka share group ingestion"
+sidebar_label: "Kafka share group ingestion"
+description: "Queue-semantics ingestion from Apache Kafka using share groups (KIP-932). Scale consumers beyond partition count with at-least-once delivery."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+Requires Apache Kafka 4.0 or higher with share groups (KIP-932) enabled on the broker.
+:::
+
+## Overview
+
+Kafka share groups (KIP-932) let multiple consumers read from the same partition concurrently. The broker manages per-record acquisition locks and explicit acknowledgement, so consumer count is not capped by partition count, joining or leaving consumers does not pause the group, and a slow record does not block its partition.
+
+Druid's `ShareGroupIndexTask` consumes from a share group and publishes segments with at-least-once delivery: records are acknowledged only after their segments are atomically registered in the metadata store.
+
+## When to use share group ingestion
+
+| Scenario | Consumer group | Share group |
+|----------|---------------|-------------|
+| Workers needed exceed partition count | Idle workers | All workers active |
+| Elastic scaling (auto-scale events) | Rebalancing pause (30-60s) | Zero pause |
+| Per-message processing time varies | Head-of-line blocking | Independent processing |
+| Ordered processing required per partition | Yes | No (delivery order not guaranteed) |
+
+Choose share groups when throughput and elastic scaling matter more than strict per-partition ordering.
+
+## Task spec
+
+Submit a `ShareGroupIndexTask` to the Overlord. There are no start/end offsets -- the broker tracks them.
+
+```json
+{
+ "type": "index_kafka_share_group",
+ "dataSchema": {
+ "dataSource": "my_datasource",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "useSchemaDiscovery": true
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": "NONE"
+ }
+ },
+ "ioConfig": {
+ "topic": "my_topic",
+ "groupId": "druid-share-group",
+ "consumerProperties": {
+ "bootstrap.servers": "kafka-broker:9092"
+ },
+ "inputFormat": {
+ "type": "json"
+ },
+ "pollTimeout": 2000
+ },
+ "tuningConfig": {
+ "type": "KafkaTuningConfig",
+ "maxRowsPerSegment": 5000000
+ }
+}
+```
+
+## IO configuration
+
+| Property | Type | Required | Default | Description |
+|----------|------|----------|---------|-------------|
+| `topic` | String | Yes | -- | Kafka topic to consume from. |
+| `groupId` | String | Yes | -- | Share group identifier. Multiple tasks with the same `groupId` share the workload. |
+| `consumerProperties` | Map | Yes | -- | Kafka consumer properties. Must include `bootstrap.servers`. See [Consumer property restrictions](#consumer-property-restrictions). |
+| `inputFormat` | Object | Yes | -- | Input format for parsing records (json, csv, avro, etc.). |
+| `pollTimeout` | Long | No | 2000 | Poll timeout in milliseconds. |
+
+### Consumer property restrictions
+
+Share consumers (KIP-932) reject some keys that are valid for regular consumer groups. Druid silently strips the keys below from `consumerProperties` (with a `WARN` log per stripped key) before constructing the `KafkaShareConsumer`:
+
+| Stripped key | Why |
+|--------------|-----|
+| `auto.offset.reset` | Initial position is broker-controlled for share groups. |
+| `enable.auto.commit` | Share consumers always require explicit `acknowledge()` + `commitSync()`. |
+| `group.instance.id` | Share groups do not support static membership. |
+| `isolation.level` | Always read-committed for share groups. |
+| `partition.assignment.strategy` | Broker controls per-record delivery for share groups. |
+| `interceptor.classes` | Not supported for share consumers. |
+| `session.timeout.ms` | Share groups have no consumer-group session model. |
+| `heartbeat.interval.ms` | Share groups have no heartbeat. |
+| `group.protocol` | Always `SHARE` for share consumers. |
+| `group.remote.assignor` | Not applicable to share groups. |
+
+`share.acknowledgement.mode=explicit` is set automatically and must not be overridden.
+
+### Tuning configuration
+
+`tuningConfig` accepts the standard `KafkaTuningConfig` fields. The runner currently honors:
+
+- `maxRowsInMemory` / `maxBytesInMemory`: triggers a mid-batch persist when the appenderator signals `isPersistRequired`.
+- `maxRowsPerSegment`: when reached during a batch the runner logs the event; over-threshold segments are pushed at the end-of-batch publish boundary.
+
+Mid-batch checkpoint and sequence rollover are not supported.
+
+## How it works
+
+1. The task subscribes to the topic with a `KafkaShareConsumer` using the configured `groupId`.
+2. The broker delivers batches of records with per-record acquisition locks.
+3. Each polled record is parsed by `StreamChunkReader` (the same multi-row parser as `KafkaIndexTask`); a record may produce zero, one, or many `InputRow`s. All resulting rows are added to the appenderator before the record is acknowledged.
+4. Parse failures go through `ParseExceptionHandler` (so `maxParseExceptions` is honored). Bytes/processed/unparseable counters are incremented exactly once per row.
+5. Segments persist mid-batch on memory pressure and unconditionally at end-of-batch, then publish atomically via `SegmentTransactionalAppendAction`.
+6. After a successful publish, every offset in the batch is acknowledged with `ACCEPT` and a `commitSync()` flushes acknowledgements to the broker.
+7. On task failure or graceful stop before publish, unacknowledged records are redelivered by the broker after the acquisition lock expires.
+
+## Safety invariants
+
+1. **ACK after publish:** `ACCEPT` is sent only after the segment is registered in the metadata store. No data loss on task failure.
+2. **Multi-row safe:** every row produced from a record is added to the appenderator before that record is acknowledged.
+3. **Resource safe:** `Appenderator` and `KafkaShareConsumer` are released on every exit path.
+4. **Terminal state:** every polled record reaches exactly one terminal state -- `ACCEPT`, `RELEASE`, or broker redelivery after lock expiry.
+
+## Graceful stop
+
+When the Overlord asks a task to stop, the runner calls `KafkaShareConsumer.wakeup()`. The in-flight `poll()` throws `WakeupException`; the runner exits the loop after committing any in-flight batch. Records polled but not yet published remain unacknowledged and are redelivered by the broker after the acquisition lock expires.
+
+## Acquisition lock duration
+
+The broker controls the lock via `group.share.record.lock.duration.ms`. The runner logs the effective value once after the first poll:
+
+```
+Effective broker acquisition lock timeout for share-group[my-group]: 30000 ms
+```
+
+A single thread does both poll and publish. If a batch exceeds the lock duration, in-flight records may be redelivered (duplicates). Tune `pollTimeout`, `maxRowsInMemory`, and `maxRowsPerSegment` so each cycle stays well under the lock window.
+
+## Scaling
+
+Tasks with the same `groupId` share the workload automatically; you can run more tasks than partitions:
+
+```
+Topic: 4 partitions
+Tasks with same groupId: 20
+Result: All 20 tasks actively consuming (broker distributes records)
+```
+
+Adding or removing tasks does not trigger a rebalancing pause.
+
+## Delivery semantics
+
+At-least-once. On task failure, records between the last committed acknowledgement and the failure point are redelivered, which may produce duplicates across restarts. A deduplication cache is planned.
+
+## Metrics
+
+In addition to the standard ingestion metrics (`ingest/events/processed`, `ingest/events/unparseable`, `ingest/persists/count`, etc.), share-group ingestion emits:
+
+| Metric | Description |
+|--------|-------------|
+| `ingest/shareGroup/commitFailures` | Per-batch count of partitions whose `commitSync()` failed. A non-zero value means the affected records will be redelivered; alert on sustained non-zero values. |
+
+## Limitations (current release)
+
+- Single-threaded ingestion per task; a future enhancement may add a background `RENEW` thread to extend the broker lock for long-running batches.
+- No supervisor integration; tasks are submitted manually via the Overlord API. A `KafkaShareGroupSupervisor` is planned as a future enhancement.
+- No deduplication cache (at-least-once).
+- Delivery order within a partition is not guaranteed.
+- Mid-batch checkpoint / sequence rollover is not supported. If a batch grossly exceeds `maxRowsPerSegment` the runner still publishes correctly (multiple segments per batch), but the threshold is only checked at end-of-batch boundaries.
+
+## Demo: end-to-end validation with Druid UI
+
+### Prerequisites
+
+- Java 17
+- Kafka 4.2.0 (with share groups enabled)
+- Druid checked out from this repository (built from source)
+
+### Step 1: Start Kafka with share groups
+
+```bash
+cd kafka_2.13-4.2.0
+
+KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
+bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
+
+echo "group.share.enable=true" >> config/server.properties
+echo "group.share.record.lock.duration.ms=30000" >> config/server.properties
+
+bin/kafka-server-start.sh config/server.properties
+```
+
+### Step 2: Create topic and produce messages
+
+```bash
+cd kafka_2.13-4.2.0
+
+bin/kafka-topics.sh --create --topic druid-share-test --partitions 4 --bootstrap-server localhost:9092
+
+bin/kafka-console-producer.sh --topic druid-share-test --bootstrap-server localhost:9092
Review Comment:
[P2] Demo produces records before the share group can read them
The demo creates and produces the sample messages before submitting druid-demo-share-group, but share-group starting position is broker-controlled and this PR strips auto.offset.reset; the tests explicitly set share.auto.offset.reset=earliest for this reason. With the broker default of latest, users following the demo will submit the task after the sample records already exist and ingest zero rows. Move producing after task startup or add a kafka-configs/AdminClient step that sets share.auto.offset.reset=earliest for the demo group before producing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [DRAFT] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3087502623
##########
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/simulate/EmbeddedShareGroupIngestionTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka.simulate;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * End-to-end integration test for Kafka share group ingestion.
+ *
+ * Demonstrates the full workflow:
+ * 1. Start an embedded Druid cluster with Kafka extension
+ * 2. Create a Kafka topic and produce CSV records
+ * 3. Submit a ShareGroupIndexTask via the Overlord API
+ * 4. Wait for records to be ingested
+ * 5. Verify data via SQL query
+ *
+ * Requires Docker for Testcontainers Kafka broker.
+ */
+public class EmbeddedShareGroupIngestionTest extends EmbeddedClusterTestBase
+{
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+
+ return cluster;
+ }
+
+ /**
+ * Showcase test: Submit a ShareGroupIndexTask, ingest data from Kafka
+ * using share group semantics, and verify via SQL.
+ */
+ @Test
+ public void test_shareGroupIngestion_basicEndToEnd()
+ {
+ final String topic = dataSource + "_topic";
+ kafkaServer.createTopicWithPartitions(topic, 2);
+
+ final int numRecords = 10;
+ kafkaServer.produceRecordsToTopic(
+ generateRecords(topic, numRecords, DateTimes.of("2025-06-01"))
+ );
+
+ final DataSchema dataSchema = DataSchema.builder()
+ .withDataSource(dataSource)
+ .withTimestamp(new TimestampSpec(COL_TIMESTAMP, null, null))
+ .withDimensions(DimensionsSpec.EMPTY)
+ .withGranularity(new UniformGranularitySpec(
+ Granularities.DAY,
+ Granularities.NONE,
+ null
+ ))
+ .build();
+
+ final ShareGroupIndexTaskIOConfig ioConfig = new ShareGroupIndexTaskIOConfig(
+ topic,
+ "druid-share-group-test",
+ kafkaServer.consumerProperties(),
+ new CsvInputFormat(List.of(COL_TIMESTAMP, COL_ITEM, COL_VALUE), null, null, false, 0, false),
+ null
+ );
+
+ final KafkaIndexTaskTuningConfig tuningConfig = new KafkaIndexTaskTuningConfig(
+ null, null, null, null, null, null, null, null, null, null,
+ null, null, null, null, null, null, null, null, null, null,
+ null, null
+ );
+
+ final ObjectMapper mapper = new DefaultObjectMapper();
+ final ShareGroupIndexTask task = new ShareGroupIndexTask(
+ null,
+ null,
+ dataSchema,
+ tuningConfig,
+ ioConfig,
+ null,
+ mapper
+ );
+
+ cluster.callApi().submitTask(task);
+
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(numRecords)
+ );
+
+ Assertions.assertEquals(
Review Comment:
Making sure we read all the records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3273986227
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/ShareGroupIndexerCrashIT.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.testing.embedded.indexing;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.indexer.granularity.UniformGranularitySpec;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskModule;
+import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTask;
+import org.apache.druid.indexing.kafka.ShareGroupIndexTaskIOConfig;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.testing.embedded.EmbeddedBroker;
+import org.apache.druid.testing.embedded.EmbeddedCoordinator;
+import org.apache.druid.testing.embedded.EmbeddedDruidCluster;
+import org.apache.druid.testing.embedded.EmbeddedHistorical;
+import org.apache.druid.testing.embedded.EmbeddedIndexer;
+import org.apache.druid.testing.embedded.EmbeddedOverlord;
+import org.apache.druid.testing.embedded.junit5.EmbeddedClusterTestBase;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Verifies at-least-once semantics: uncommitted records are redelivered after Indexer crash.
+ */
+public class ShareGroupIndexerCrashIT extends EmbeddedClusterTestBase
+{
+ private static final long SHARE_CONSUMER_READY_DELAY_MS = 3_000L;
+ private static final String COL_TIMESTAMP = "__time";
+ private static final String COL_ITEM = "item";
+ private static final String COL_VALUE = "value";
+ private static final String GROUP_ID = "crash-it-group";
+
+ private final EmbeddedCoordinator coordinator = new EmbeddedCoordinator();
+ private final EmbeddedOverlord overlord = new EmbeddedOverlord();
+ private final EmbeddedIndexer indexer = new EmbeddedIndexer();
+ private final EmbeddedHistorical historical = new EmbeddedHistorical();
+ private final EmbeddedBroker broker = new EmbeddedBroker();
+
+ private ShareGroupKafkaResource kafkaServer;
+ private final ObjectMapper mapper = new DefaultObjectMapper();
+
+ @Override
+ public EmbeddedDruidCluster createCluster()
+ {
+ kafkaServer = new ShareGroupKafkaResource();
+ final EmbeddedDruidCluster cluster = EmbeddedDruidCluster.withEmbeddedDerbyAndZookeeper();
+ indexer.addProperty("druid.segment.handoff.pollDuration", "PT0.1s");
+ cluster.addExtension(KafkaIndexTaskModule.class)
+ .addResource(kafkaServer)
+ .useLatchableEmitter()
+ .useDefaultTimeoutForLatchableEmitter(60)
+ .addCommonProperty("druid.monitoring.emissionPeriod", "PT0.1s")
+ .addServer(coordinator)
+ .addServer(overlord)
+ .addServer(indexer)
+ .addServer(historical)
+ .addServer(broker);
+ return cluster;
+ }
+
+ @Test
+ public void test_indexerCrash_midIngestion_noDataLossAfterRecovery() throws Exception
+ {
+ final String topic = dataSource + "_crash_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset(GROUP_ID, "earliest");
+
+ // Produce all records before starting task so Indexer is polling them when it crashes.
+ final int batchA = 20;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(batchA, 0, "2025-09-01"));
+
+ // Task 1: starts ingesting but Indexer crashes before it completes.
+ final String taskId1 = submitTask(topic);
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Wait for at least some rows to be processed (but not all published to segments).
+ indexer.latchableEmitter().waitForEventAggregate(
+ event -> event.hasMetricName("ingest/events/processed")
+ .hasDimension(DruidMetrics.DATASOURCE, dataSource),
+ agg -> agg.hasSumAtLeast(1)
+ );
+
+ // Simulate JVM crash: stop the indexer abruptly (no graceful shutdown).
+ indexer.stop();
+
+ // Restart the indexer.
+ indexer.start();
+
+ // Task 2: same group-id. Broker redelivers any unacknowledged records.
+ final String taskId2 = submitTask(topic);
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Produce a second batch to ensure Task 2 is actively consuming.
+ final int batchB = 10;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(batchB, batchA, "2025-09-02"));
+
+ // Give Task 2 time to consume redelivered + new records.
+ // Post-restart emitter has a fresh counter, so we use a dwell + SQL check instead.
+ Thread.sleep(15_000L);
+
+ cluster.callApi().onLeaderOverlord(o -> o.cancelTask(taskId2));
+ cluster.callApi().waitForTaskToFinish(taskId2, overlord.latchableEmitter());
+
+ cluster.callApi().waitForAllSegmentsToBeAvailable(dataSource, coordinator, broker);
+
+ // At-least-once guarantee: row count >= batchA + batchB (no loss); may be slightly higher due to redelivery.
+ final long rowCount = Long.parseLong(cluster.runSql("SELECT COUNT(*) FROM %s", dataSource));
+ Assertions.assertTrue(
+ rowCount >= batchA + batchB,
+ "Expected at least [" + (batchA + batchB) + "] rows but got [" + rowCount + "]"
+ );
+ }
+
+ @Test
+ public void test_indexerCrash_beforeAnyAck_allRecordsRedelivered() throws Exception
+ {
+ final String topic = dataSource + "_crash_no_ack_topic";
+ kafkaServer.createTopicWithPartitions(topic, 1);
+ kafkaServer.setShareGroupAutoOffsetReset(GROUP_ID + "-noack", "earliest");
+
+ final int numRecords = 10;
+ kafkaServer.publishRecordsToTopic(topic, csvRecords(numRecords, 0, "2025-10-01"));
+
+ // Task 1: starts but is killed before its first publish (no ack committed).
+ submitTask(topic, GROUP_ID + "-noack");
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ indexer.stop();
+ indexer.start();
+
+ // Task 2 (same group): broker must redeliver all numRecords because none were acked.
+ final String taskId2 = submitTask(topic, GROUP_ID + "-noack");
+ Thread.sleep(SHARE_CONSUMER_READY_DELAY_MS);
+
+ // Give Task 2 time to consume redelivered records before cancel.
+ Thread.sleep(15_000L);
Review Comment:
[P2] Crash tests no longer prove redelivery
Replacing the post-restart waitForEventAggregate checks with fixed sleeps means these crash tests no longer assert that Task 2 consumed any redelivered records. Task 1 is allowed to run for 3 seconds before indexer.stop(), and this runner publishes and acks every batch, so it can already make the final SQL row count pass; in the first test the new batch can also satisfy the count. A regression that fails to redeliver unacknowledged records could now slip through. Please wait on a post-restart ingest signal or otherwise assert rows were processed by the restarted task before cancelling it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3288578435
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,497 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Supplier;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.DynamicPartitionsSpec;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.task.InputRowFilter;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.StreamChunkReader;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+import org.apache.kafka.common.errors.WakeupException;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+
+/**
+ * Single-threaded ingestion loop for {@link ShareGroupIndexTask}. Records are
+ * {@code ACCEPT}-acknowledged only after the containing segment is published;
+ * any earlier failure leaves them unacknowledged so the broker redelivers
+ * them once the acquisition lock expires.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ static final String METRIC_COMMIT_FAILURES = "ingest/shareGroup/commitFailures";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+ private final Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory;
+
+ private final AtomicReference<AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>>
+ activeSupplier = new AtomicReference<>();
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this(task, toolbox, configMapper, null);
+ }
+
+ @VisibleForTesting
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper,
+ @Nullable Function<ShareGroupIndexTaskIOConfig,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity>> supplierFactory
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ this.supplierFactory = supplierFactory != null ? supplierFactory : this::createDefaultRecordSupplier;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ // Raw type mirrors SeekableStreamIndexTaskRunner; works around
+ // OrderedPartitionableRecord.getData() returning a wildcard list.
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ final StreamChunkReader chunkReader = new StreamChunkReader<KafkaRecordEntity>(
+ inputFormat,
+ inputRowSchema,
+ dataSchema.getTransformSpec(),
+ toolbox.getIndexingTmpDir(),
+ InputRowFilter.allowAll(),
+ rowIngestionMeters,
+ parseExceptionHandler
+ );
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ final org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor metricsMonitor =
+ new org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor(
+ segmentGenerationMetrics,
+ rowIngestionMeters,
+ task.getMetricBuilder()
+ );
+ toolbox.addMonitor(metricsMonitor);
+
+ boolean runLoopCompleted = false;
+ try (final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier =
+ supplierFactory.apply(ioConfig)) {
+ activeSupplier.set(recordSupplier);
+ final TaskStatus status = runLoop(driver, appenderator, recordSupplier, chunkReader, ioConfig, tuningConfig, toolbox);
+ runLoopCompleted = true;
+ return status;
+ }
+ finally {
+ activeSupplier.set(null);
+ if (!runLoopCompleted) {
+ try {
+ appenderator.closeNow();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception during emergency closeNow() of Appenderator in run(); continuing teardown.");
+ }
+ }
+ try {
+ toolbox.removeMonitor(metricsMonitor);
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception removing TaskRealtimeMetricsMonitor; continuing teardown.");
+ }
+ try {
+ driver.close();
+ }
+ catch (Exception e) {
+ log.warn(e, "Exception closing StreamAppenderatorDriver; continuing teardown.");
+ }
+ }
+ }
+
+ @VisibleForTesting
+ TaskStatus runLoop(
+ StreamAppenderatorDriver driver,
+ Appenderator appenderator,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ StreamChunkReader chunkReader,
+ ShareGroupIndexTaskIOConfig ioConfig,
+ KafkaIndexTaskTuningConfig tuningConfig,
+ TaskToolbox toolbox
+ ) throws Exception
+ {
+ // Share groups manage delivery state on the broker; the Committer is a
+ // no-op placeholder required by the appenderator driver contract.
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+ boolean lockTimeoutLogged = false;
+ boolean appenderatorClosedNormally = false;
+
+ try {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+ driver.startJob(segmentId -> true);
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records;
+ try {
+ records = recordSupplier.poll(ioConfig.getPollTimeout());
+ }
+ catch (WakeupException e) {
+ if (task.isStopRequested()) {
+ log.info("Wake-up received while stopGracefully is in progress; exiting ingestion loop.");
+ break;
+ }
+ throw e;
+ }
+
+ if (!lockTimeoutLogged) {
+ final Optional<Integer> effectiveLockMs = recordSupplier.acquisitionLockTimeoutMs();
+ log.info(
+ "Effective broker acquisition lock timeout for share-group[%s]: %s ms",
+ ioConfig.getGroupId(),
+ effectiveLockMs.map(String::valueOf).orElse("<unknown>")
+ );
+ lockTimeoutLogged = true;
+ }
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+ boolean midBatchPersistNeeded = false;
+ boolean pushThresholdLogged = false;
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ final List<InputRow> rows = chunkReader.parse(record.getData(), false);
+
+ for (InputRow row : rows) {
+ final AppenderatorDriverAddResult addResult = driver.add(
+ row,
+ SEQUENCE_NAME,
Review Comment:
[P1] Use per-task sequence names for segment allocation
Every `ShareGroupIndexTaskRunner` uses the same hard-coded sequence name when adding rows. Druid segment allocation is idempotent for the same `sequenceName`, and this call also skips lineage checks, so concurrent share-group tasks ingesting the same datasource interval can be handed the same pending segment id. If one task publishes that segment first, another task can treat the id as already published and then acknowledge its Kafka records even though its rows were not registered. Make the allocation sequence unique per task or otherwise isolate concurrent producers before supporting multiple tasks in one share group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1 - Draft] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r3207056507
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupIndexTaskRunner.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.data.input.Committer;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.ByteEntity;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexing.appenderator.ActionBasedPublishedSegmentRetriever;
+import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.TaskLockType;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.TaskLocks;
+import org.apache.druid.indexing.common.actions.LockReleaseAction;
+import org.apache.druid.indexing.common.actions.SegmentAllocateAction;
+import org.apache.druid.indexing.common.actions.SegmentTransactionalAppendAction;
+import org.apache.druid.indexing.common.task.Tasks;
+import org.apache.druid.indexing.input.InputRowSchemas;
+import org.apache.druid.indexing.overlord.SegmentPublishResult;
+import org.apache.druid.indexing.seekablestream.SeekableStreamAppenderatorConfig;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.common.config.Configs;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+import org.apache.druid.segment.SegmentSchemaMapping;
+import org.apache.druid.segment.incremental.ParseExceptionHandler;
+import org.apache.druid.segment.incremental.RowIngestionMeters;
+import org.apache.druid.segment.indexing.DataSchema;
+import org.apache.druid.segment.realtime.SegmentGenerationMetrics;
+import org.apache.druid.segment.realtime.appenderator.Appenderator;
+import org.apache.druid.segment.realtime.appenderator.AppenderatorDriverAddResult;
+import org.apache.druid.segment.realtime.appenderator.SegmentsAndCommitMetadata;
+import org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver;
+import org.apache.druid.segment.realtime.appenderator.TransactionalSegmentPublisher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedPartialShardSpec;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.common.base.Supplier;
+
+/**
+ * Ingestion loop for {@link ShareGroupIndexTask}.
+ *
+ * Phase 1: single-threaded. Polls from KafkaShareConsumer, parses rows,
+ * adds to appenderator, persists, publishes segments, then ACKs records
+ * and commits. Records are only acknowledged AFTER segment publication
+ * succeeds (INVARIANT 1).
+ *
+ * On failure, unacknowledged records are redelivered by the broker.
+ */
+public class ShareGroupIndexTaskRunner
+{
+ private static final Logger log = new Logger(ShareGroupIndexTaskRunner.class);
+ private static final String SEQUENCE_NAME = "share_group_seq_0";
+
+ private final ShareGroupIndexTask task;
+ private final TaskToolbox toolbox;
+ private final ObjectMapper configMapper;
+
+ ShareGroupIndexTaskRunner(
+ ShareGroupIndexTask task,
+ TaskToolbox toolbox,
+ ObjectMapper configMapper
+ )
+ {
+ this.task = task;
+ this.toolbox = toolbox;
+ this.configMapper = configMapper;
+ }
+
+ public TaskStatus run() throws Exception
+ {
+ final DataSchema dataSchema = task.getDataSchema();
+ final KafkaIndexTaskTuningConfig tuningConfig = task.getTuningConfig();
+ final ShareGroupIndexTaskIOConfig ioConfig = task.getIOConfig();
+
+ final RowIngestionMeters rowIngestionMeters =
+ toolbox.getRowIngestionMetersFactory().createRowIngestionMeters();
+ final ParseExceptionHandler parseExceptionHandler = new ParseExceptionHandler(
+ rowIngestionMeters,
+ tuningConfig.isLogParseExceptions(),
+ tuningConfig.getMaxParseExceptions(),
+ tuningConfig.getMaxSavedParseExceptions()
+ );
+ final SegmentGenerationMetrics segmentGenerationMetrics = new SegmentGenerationMetrics();
+
+ final InputRowSchema inputRowSchema = InputRowSchemas.fromDataSchema(dataSchema);
+ final InputFormat inputFormat = ioConfig.getInputFormat();
+ if (inputFormat == null) {
+ throw new ISE("inputFormat must be specified in ioConfig");
+ }
+
+ final LockGranularity lockGranularity = Configs.valueOrDefault(
+ task.getContextValue(Tasks.FORCE_TIME_CHUNK_LOCK_KEY, Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK),
+ Tasks.DEFAULT_FORCE_TIME_CHUNK_LOCK
+ )
+ ? LockGranularity.TIME_CHUNK
+ : LockGranularity.SEGMENT;
+ final TaskLockType lockType = TaskLocks.determineLockTypeForAppend(task.getContext());
+
+ // Create appenderator and driver
+ final Appenderator appenderator = toolbox.getAppenderatorsManager().createRealtimeAppenderatorForTask(
+ toolbox.getSegmentLoaderConfig(),
+ task.getId(),
+ dataSchema,
+ SeekableStreamAppenderatorConfig.fromTuningConfig(
+ tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
+ toolbox.getProcessingConfig()
+ ),
+ toolbox.getConfig(),
+ segmentGenerationMetrics,
+ toolbox.getSegmentPusher(),
+ toolbox.getJsonMapper(),
+ toolbox.getIndexIO(),
+ toolbox.getIndexMerger(),
+ toolbox.getQueryRunnerFactoryConglomerate(),
+ toolbox.getSegmentAnnouncer(),
+ toolbox.getEmitter(),
+ toolbox.getQueryProcessingPool(),
+ toolbox.getJoinableFactory(),
+ toolbox.getCache(),
+ toolbox.getCacheConfig(),
+ toolbox.getCachePopulatorStats(),
+ toolbox.getPolicyEnforcer(),
+ rowIngestionMeters,
+ parseExceptionHandler,
+ toolbox.getCentralizedTableSchemaConfig(),
+ interval -> {
+ toolbox.getTaskActionClient().submit(new LockReleaseAction(interval));
+ }
+ );
+
+ final StreamAppenderatorDriver driver = new StreamAppenderatorDriver(
+ appenderator,
+ new ActionBasedSegmentAllocator(
+ toolbox.getTaskActionClient(),
+ dataSchema,
+ (schema, row, sequenceName, previousSegmentId, skipSegmentLineageCheck) -> new SegmentAllocateAction(
+ schema.getDataSource(),
+ row.getTimestamp(),
+ schema.getGranularitySpec().getQueryGranularity(),
+ schema.getGranularitySpec().getSegmentGranularity(),
+ sequenceName,
+ previousSegmentId,
+ skipSegmentLineageCheck,
+ NumberedPartialShardSpec.instance(),
+ lockGranularity,
+ lockType
+ )
+ ),
+ toolbox.getSegmentHandoffNotifierFactory(),
+ new ActionBasedPublishedSegmentRetriever(toolbox.getTaskActionClient()),
+ toolbox.getDataSegmentKiller(),
+ toolbox.getJsonMapper(),
+ segmentGenerationMetrics
+ );
+
+ try (final KafkaShareGroupRecordSupplier recordSupplier = createRecordSupplier(ioConfig)) {
+ recordSupplier.subscribe(Collections.singleton(ioConfig.getTopic()));
+
+ driver.startJob(segmentId -> true);
+
+ final Supplier<Committer> committerSupplier = () -> new Committer()
+ {
+ @Override
+ public Object getMetadata()
+ {
+ return ImmutableMap.of("type", "share_group");
+ }
+
+ @Override
+ public void run()
+ {
+ // no-op: share group does not need client-side offset persistence
+ }
+ };
+
+ log.info("Starting share group ingestion loop for topic[%s], group[%s].",
+ ioConfig.getTopic(), ioConfig.getGroupId());
+
+ long totalRowsIngested = 0;
+
+ while (!task.isStopRequested()) {
+
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(ioConfig.getPollTimeout());
+
+ if (records.isEmpty()) {
+ continue;
+ }
+
+ // Track offsets of records in this batch for acknowledgement after publish
+ final Map<KafkaTopicPartition, List<Long>> batchOffsets = new HashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ batchOffsets.computeIfAbsent(record.getPartitionId(), k -> new ArrayList<>())
+ .add(record.getSequenceNumber());
+
+ if (record.getData() == null) {
+ continue;
+ }
+
+ for (ByteEntity entity : record.getData()) {
+ try {
+ final InputRow row = parseRow(entity, inputFormat, inputRowSchema, toolbox);
Review Comment:
Updated for multi row : https://github.com/apache/druid/pull/19311/changes#r3207054221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4022220720
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/common/AcknowledgingRecordSupplier.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.seekablestream.common;
+
+import org.apache.druid.data.input.impl.ByteEntity;
+
+import javax.validation.constraints.NotNull;
Review Comment:
Thanks for checking https://github.com/apache/druid/pull/19311/commits/fd1b9af8e1da39f24eefac9a1d3a8e0d24fb2c8d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: [Phase 1] Queue Semantics support in Kafka Ingestion (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #19311:
URL: https://github.com/apache/druid/pull/19311#discussion_r4132997524
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupBatchStore.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.druid.storage.StorageConnector;
+
+import java.io.FilterOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.DigestInputStream;
+import java.security.DigestOutputStream;
+import java.security.MessageDigest;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.UUID;
+
+final class ShareGroupBatchStore
+{
+ static final class StoredBatch
+ {
+ private final String path;
+ private final String sha256;
+ private final long size;
+ private final ShareGroupRawBatch batch;
+
+ private StoredBatch(String path, String sha256, long size, ShareGroupRawBatch batch)
+ {
+ this.path = path;
+ this.sha256 = sha256;
+ this.size = size;
+ this.batch = batch;
+ }
+
+ String getPath()
+ {
+ return path;
+ }
+
+ String getSha256()
+ {
+ return sha256;
+ }
+
+ long getSize()
+ {
+ return size;
+ }
+
+ ShareGroupRawBatch getBatch()
+ {
+ return batch;
+ }
+ }
+
+ private static final class CountingOutputStream extends FilterOutputStream
+ {
+ private long count;
+
+ private CountingOutputStream(OutputStream outputStream)
+ {
+ super(outputStream);
+ }
+
+ @Override
+ public void write(int value) throws IOException
+ {
+ out.write(value);
+ count++;
+ }
+
+ @Override
+ public void write(byte[] bytes, int offset, int length) throws IOException
+ {
+ out.write(bytes, offset, length);
+ count += length;
+ }
+
+ private long getCount()
+ {
+ return count;
+ }
+ }
+
+ private final StorageConnector storageConnector;
+ private final String taskId;
+ private final ShareGroupRawBatchCodec codec = new ShareGroupRawBatchCodec();
+
+ ShareGroupBatchStore(StorageConnector storageConnector, String taskId)
+ {
+ this.storageConnector = Preconditions.checkNotNull(storageConnector, "storageConnector");
+ this.taskId = Preconditions.checkNotNull(taskId, "taskId");
+ }
+
+ StoredBatch store(
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records
+ ) throws IOException
+ {
+ return store(ShareGroupRawBatch.fromOrderedRecords(records));
+ }
+
+ StoredBatch store(ShareGroupRawBatch batch) throws IOException
+ {
+ final String path = objectPath(batch);
+ final MessageDigest writeDigest = ShareGroupRawBatchCodec.newDigest();
+ final CountingOutputStream countingOutput;
+ try (OutputStream storageOutput = storageConnector.write(path)) {
+ countingOutput = new CountingOutputStream(storageOutput);
+ codec.write(batch, new DigestOutputStream(countingOutput, writeDigest));
Review Comment:
## CodeQL / Potential output resource leak
This DigestOutputStream is not always closed on method exit.
[Show more details](https://github.com/apache/druid/security/code-scanning/12032)
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/ShareGroupAcquisitionLoop.java:
##########
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.kafka;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.data.input.kafka.KafkaRecordEntity;
+import org.apache.druid.data.input.kafka.KafkaTopicPartition;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgeType;
+import org.apache.druid.indexing.seekablestream.common.AcknowledgingRecordSupplier;
+import org.apache.druid.indexing.seekablestream.common.OrderedPartitionableRecord;
+import org.apache.kafka.common.errors.WakeupException;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executor;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.ToLongFunction;
+
+final class ShareGroupAcquisitionLoop
+{
+ private static final class StagingCompletion
+ {
+ private static final StagingCompletion STOP = new StagingCompletion(List.of(), false, true);
+
+ private final List<ShareGroupAcquisitionRegistry.UploadAttempt> attempts;
+ private final boolean successful;
+ private final boolean stop;
+
+ private StagingCompletion(
+ List<ShareGroupAcquisitionRegistry.UploadAttempt> attempts,
+ boolean successful,
+ boolean stop
+ )
+ {
+ this.attempts = List.copyOf(attempts);
+ this.successful = successful;
+ this.stop = stop;
+ }
+
+ private static StagingCompletion success(List<ShareGroupAcquisitionRegistry.UploadAttempt> attempts)
+ {
+ return new StagingCompletion(attempts, true, false);
+ }
+
+ private static StagingCompletion failure(List<ShareGroupAcquisitionRegistry.UploadAttempt> attempts)
+ {
+ return new StagingCompletion(attempts, false, false);
+ }
+ }
+
+ private final String topic;
+ private final AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier;
+ private final ShareGroupBatchStager batchStager;
+ private final Executor stagingExecutor;
+ private final ShareGroupAcquisitionRegistry registry;
+ private final ToLongFunction<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> byteEstimator;
+ private final long pollTimeoutMs;
+ private final long maxPollIntervalMs;
+ private final BlockingQueue<StagingCompletion> completions = new LinkedBlockingQueue<>();
+ private final AtomicBoolean stopRequested = new AtomicBoolean();
+ private Thread ownerThread;
+
+ ShareGroupAcquisitionLoop(
+ String topic,
+ AcknowledgingRecordSupplier<KafkaTopicPartition, Long, KafkaRecordEntity> recordSupplier,
+ ShareGroupBatchStager batchStager,
+ Executor stagingExecutor,
+ ShareGroupAcquisitionRegistry registry,
+ ToLongFunction<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> byteEstimator,
+ long pollTimeoutMs,
+ long maxPollIntervalMs
+ )
+ {
+ Preconditions.checkArgument(pollTimeoutMs >= 0, "pollTimeoutMs must not be negative");
+ Preconditions.checkArgument(maxPollIntervalMs > 0, "maxPollIntervalMs must be positive");
+ this.topic = Preconditions.checkNotNull(topic, "topic");
+ this.recordSupplier = Preconditions.checkNotNull(recordSupplier, "recordSupplier");
+ this.batchStager = Preconditions.checkNotNull(batchStager, "batchStager");
+ this.stagingExecutor = Preconditions.checkNotNull(stagingExecutor, "stagingExecutor");
+ this.registry = Preconditions.checkNotNull(registry, "registry");
+ this.byteEstimator = Preconditions.checkNotNull(byteEstimator, "byteEstimator");
+ this.pollTimeoutMs = pollTimeoutMs;
+ this.maxPollIntervalMs = maxPollIntervalMs;
+ }
+
+ void run() throws Exception
+ {
+ checkOwnerThread();
+ recordSupplier.subscribe(Set.of(topic));
+ while (!stopRequested.get()) {
+ try {
+ runOnce();
+ }
+ catch (WakeupException e) {
+ if (!stopRequested.get()) {
+ throw e;
+ }
+ }
+ }
+ }
+
+ void runOnce() throws InterruptedException
+ {
+ checkOwnerThread();
+ drainCompletions();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> records =
+ recordSupplier.poll(pollTimeoutMs);
+ drainCompletions();
+ if (records.isEmpty()) {
+ return;
+ }
+
+ final long acquisitionLockTimeoutMs = recordSupplier.acquisitionLockTimeoutMs().orElse(-1);
+ final Map<ShareGroupRecordIdentity,
+ OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> tracked = new LinkedHashMap<>();
+ final List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> saturated =
+ new ArrayList<>();
+ final Map<KafkaTopicPartition,
+ List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>>> newRecordsByPartition =
+ new LinkedHashMap<>();
+
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : records) {
+ final ShareGroupRecordIdentity identity = ShareGroupRecordIdentity.from(record);
+ final ShareGroupAcquisitionRegistry.Observation observation = registry.observe(
+ record,
+ byteEstimator.applyAsLong(record),
+ acquisitionLockTimeoutMs,
+ maxPollIntervalMs
+ );
+ if (observation == ShareGroupAcquisitionRegistry.Observation.SATURATED) {
+ saturated.add(record);
+ } else {
+ tracked.put(identity, record);
+ if (observation == ShareGroupAcquisitionRegistry.Observation.ADDED) {
+ newRecordsByPartition.computeIfAbsent(record.getPartitionId(), ignored -> new ArrayList<>()).add(record);
+ }
+ }
+ }
+
+ for (List<OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> partitionRecords
+ : newRecordsByPartition.values()) {
+ submit(partitionRecords);
+ }
+
+ waitForAcknowledgementDecisions(tracked.keySet());
+
+ boolean terminalAcknowledgementAssigned = false;
+ for (OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record : saturated) {
+ recordSupplier.acknowledge(
+ record.getPartitionId(),
+ record.getSequenceNumber(),
+ AcknowledgeType.RELEASE
+ );
+ terminalAcknowledgementAssigned = true;
+ }
+ for (Map.Entry<ShareGroupRecordIdentity,
+ OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity>> trackedRecord : tracked.entrySet()) {
+ final AcknowledgeType type = registry.acknowledgementFor(
+ trackedRecord.getKey(),
+ stopRequested.get()
+ ).orElseThrow(() -> new IllegalStateException("No acknowledgement decision for " + trackedRecord.getKey()));
+ final OrderedPartitionableRecord<KafkaTopicPartition, Long, KafkaRecordEntity> record = trackedRecord.getValue();
+ recordSupplier.acknowledge(record.getPartitionId(), record.getSequenceNumber(), type);
+ registry.acknowledgementAssigned(trackedRecord.getKey(), type);
+ terminalAcknowledgementAssigned |= type != AcknowledgeType.RENEW;
+ }
+
+ if (terminalAcknowledgementAssigned) {
+ registry.acknowledgementsFlushed(recordSupplier.flushAcknowledgementsSync());
+ }
+ }
+
+ void requestStop()
+ {
+ stopRequested.set(true);
+ completions.offer(StagingCompletion.STOP);
Review Comment:
## CodeQL / Ignored error status of call
Method requestStop ignores exceptional return value of BlockingQueue<StagingCompletion>.offer.
[Show more details](https://github.com/apache/druid/security/code-scanning/12033)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org