You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "udaysagar2177 (via GitHub)" <gi...@apache.org> on 2026/02/12 09:53:17 UTC

[PR] [feature] Add Kafka microbatch ingestion plugin #17331 [pinot]

udaysagar2177 opened a new pull request, #17688:
URL: https://github.com/apache/pinot/pull/17688

   This PR introduces a new streaming ingestion pattern for Apache Pinot - MicroBatch ingestion from Kafka. Instead of processing individual records from Kafka messages, this consumer reads Kafka messages containing JSON protocol references to batch files stored in PinotFS (S3, HDFS, GCS, Azure, etc.) or inline base64-encoded data. See https://github.com/apache/pinot/issues/17331 for motivation.
   
   ### Design Overview
   
   **Wire Protocol (Version 1):**
   ```
   +----------+---------------------------+
   | version  | JSON payload bytes        |
   | (1 byte) |                           |
   +----------+---------------------------+
   ```
   
   **JSON Payload Examples:**
   ```json
   // URI type - reference to file in PinotFS
   {"type":"uri","format":"avro","uri":"s3://bucket/batch.avro","numRecords":1000}
   
   // DATA type - inline base64-encoded data
   {"type":"data","format":"avro","data":"<base64>","numRecords":100}
   ```
   
   **Key Components:**
   - `KafkaMicroBatchConsumerFactory` - Factory that creates MicroBatch consumers
   - `MicroBatchProtocol` / `MicroBatchPayloadV1` - Protocol parsing and validation
   - `MicroBatchQueueManager` - Orchestrates parallel file downloads and batch conversion
   - `MicroBatchStreamPartitionMsgOffset` - Composite offset (Kafka offset + record offset within batch) enabling mid-batch resume after segment commits
   - `MessageBatchReader` - Converts downloaded files to `MessageBatch<GenericRow>` using Pinot's RecordReader
   
   **Supported Formats:** AVRO, Parquet, JSON
   
   ### Configuration
   
   ```json
   {
     "streamConfigs": {
       "streamType": "kafka",
       "stream.kafka.topic.name": "microbatch-topic",
       "stream.kafka.broker.list": "localhost:9092",
       "stream.kafka.consumer.factory.class.name":
         "org.apache.pinot.plugin.stream.microbatch.kafka30.KafkaMicroBatchConsumerFactory",
       "stream.microbatch.kafka.file.fetch.threads": "2"
     }
   }
   ```
   
   See `CONFIGURATION.md` for complete documentation including PinotFS setup for S3/HDFS/GCS/Azure.
   
   ### Testing
   
   - **Unit tests:** `MicroBatchProtocolTest` (30+ test cases), `MicroBatchStreamPartitionMsgOffsetFactoryTest`, `MicroBatchStreamMetadataProviderTest`, `MicroBatchQueueManagerTest`
   - **Integration test:** `KafkaPartitionLevelMicroBatchConsumerTest` - Tests with real embedded Kafka, including mid-batch resume scenarios
   - **Cluster integration test:** `MicroBatchRealtimeClusterIntegrationTest` - End-to-end test with full Pinot cluster
   
   ### Compatibility Notes
   
   - This is a new plugin module - no changes to existing APIs or behavior
   - The composite offset format (`{"kmo":X,"mbro":Y}`) is specific to MicroBatch consumers and not compatible with standard Kafka consumer offsets
   - `supportsOffsetLag()` returns false since we cannot accurately estimate lag without knowing record counts in unconsumed batches. We can have it return batch counts if needed.
   
   ### Open Questions for Reviewers
   
   I’ve added TODOs in the code to highlight open questions for reviewers, please take a look.
   
   ---
   
   ## Checklist
   
   - [x] Code compiles and follows Pinot style guidelines
   - [x] Unit tests added and passing
   - [x] Integration tests added and passing
   - [x] Documentation added (`CONFIGURATION.md`)
   - [x] No new external dependencies introduced


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "udaysagar2177 (via GitHub)" <gi...@apache.org>.
udaysagar2177 commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-3893971229

   @noob-se7en Thanks for the feedback. The current Kafka-based approach remains useful for users who prefer to avoid the operational complexity of streaming commits into Apache Iceberg.
   
   Regarding reading directly from the upstream object store: clarification on the intended mechanism would help. How would Apache Pinot discover new files and track which ones have already been processed?
   
   One possible direction is using Iceberg tables, which provide snapshots, manifests with file metadata, and incremental scans for new files. The existing microbatch framework would remain largely unchanged - file discovery from Iceberg snapshots instead of Kafka, while MicroBatchQueueManager continues handling download and processing. I posted a short PEP describing this Iceberg-based approach here: https://github.com/apache/pinot/issues/17694. Could you clarify whether your suggestion refers to a different model?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-3892049578

   Should we consider enabling a consumer to read directly from the upstream object store? This could eliminate the requirement here for users to provision Kafka.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-3905049963

   > 
   > Regarding reading directly from the upstream object store: clarification on the intended mechanism would help. How would Apache Pinot discover new files and track which ones have already been processed?
   
   Idea would be around pinot to have its own checkpointing but I think the implementation will be complex, though the setup for a user will be quite simpler. 
   For this PR, Can you help in giving an example like for a user to use this feature what can the upstream setup look like?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "udaysagar2177 (via GitHub)" <gi...@apache.org>.
udaysagar2177 commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-4173313089

   @Jackie-Jiang @xiangfu0 can you please help review this PR? It has been open for a while now without any feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "udaysagar2177 (via GitHub)" <gi...@apache.org>.
udaysagar2177 commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-3925053884

   @noob-se7en In some deployments, an upstream system may already be producing Avro or Parquet files and storing them in object storage. In such cases, requiring an additional ingestion-specific export step just for Pinot can introduce extra complexity.
   
   With this microbatch framework, the upstream system can simply pass along the file metadata for those already-produced files, and Pinot can index them directly. The goal is to make it easier to integrate Pinot into existing data pipelines without forcing changes to how data is produced or stored upstream.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [feature] Add Kafka microbatch ingestion plugin [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17688:
URL: https://github.com/apache/pinot/pull/17688#issuecomment-3901925743

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17688?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 55.59%. Comparing base ([`0f93d52`](https://app.codecov.io/gh/apache/pinot/commit/0f93d52ef0bfa1c2dc367b31f137a63bd25f9d2f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e86f46f`](https://app.codecov.io/gh/apache/pinot/commit/e86f46f3d7e0109788ddc03942896d4f8e1d4ed6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 18 commits behind head on master.
   > :exclamation:  There is a different number of reports uploaded between BASE (0f93d52) and HEAD (e86f46f). Click for more details.
   > 
   > <details><summary>HEAD has 36 uploads less than BASE</summary>
   >
   >| Flag | BASE (0f93d52) | HEAD (e86f46f) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|10|1|
   >|java-11|5|0|
   >|integration|6|0|
   >|integration2|2|0|
   >|unittests2|2|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #17688      +/-   ##
   ============================================
   - Coverage     63.25%   55.59%   -7.66%     
   + Complexity     1499      720     -779     
   ============================================
     Files          3174     2476     -698     
     Lines        190373   140380   -49993     
     Branches      29089    22366    -6723     
   ============================================
   - Hits         120419    78044   -42375     
   + Misses        60610    55741    -4869     
   + Partials       9344     6595    -2749     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <ø> (-7.63%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <ø> (-7.66%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <ø> (-7.66%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <ø> (-0.04%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17688/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17688?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org