Re: [I] Kafka 4.x Queue Semantics support in Kafka Ingestion (druid)

["Shekharrajak (via GitHub)" <gi...@apache.org>]

Shekharrajak commented on issue #18439:
URL: https://github.com/apache/druid/issues/18439#issuecomment-4236351269

   
   ###  Proposed Plan
   
   *   **`KafkaShareIndexTask`:**  A new task type that `subscribe()`s to a Share Group, using the `ShareConsumer` API to poll for records across all topic partitions.
   *   **Simplified Supervisor:** The `Supervisor` will manage the number of Druid tasks, removing the need for partition assignment logic.
   *   **Consistency Model:** This is an **opt-in mode** for users that prioritize scale and throughput; strict per-partition ordering is not guaranteed.
   
   ### Proposed Configuration
   
   Users will enable Share Group ingestion in the `ioConfig`:
   
   ```json
   "ioConfig": {
     "type": "kafka",
     "useShareGroup": true,
     "numTasks": 20  // Can exceed partition count
   }
   ```
   -----
   
   
   This feature should be the default choice for Druid users when:
   
   -     Strict ordering is not required: Since multiple tasks pull from one partition, records may be processed out of sequence (perfectly fine for most time-series OLAP use cases).
   
   - Unpredictable Spikes: When ingestion traffic is bursty and requires rapid scaling.
   
   -     High Computational Cost: When Druid is doing complex flattenSpec or transformSpec operations during ingestion that slow down individual tasks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org