You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/07/09 22:32:52 UTC

[I] Fix multi-threaded access to KafkaConsumer [pinot]

Jackie-Jiang opened a new issue, #16314:
URL: https://github.com/apache/pinot/issues/16314

   `KafkaConsumer` cannot be used in a multi-threaded manner
   
   ```
   2025/07/09 22:30:06.371 WARN [RealtimeSegmentDataManager_*] [Start a Pinot [SERVER]] Cannot fetch stream offset with criteria OffsetCriteria{_offsetType=LARGEST, _offsetString='largest'} for clientId * and partitionGroupId 16 with maxWaitTime 5000
   java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
           at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2445) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2429) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:2213) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchStreamPartitionOffset(KafkaStreamMetadataProvider.java:103) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchStreamOffset(RealtimeSegmentDataManager.java:1778) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1753) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1758) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.server.starter.helix.FreshnessBasedConsumptionStatusChecker.isSegmentCaughtUp(FreshnessBasedConsumptionStatusChecker.java:76) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.server.starter.helix.IngestionBasedConsumptionStatusChecker.getNumConsumingSegmentsNotReachedIngestionCriteria(IngestionBasedConsumptionStatusChecker.java:96) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.common.utils.ServiceStatus$RealtimeConsumptionCatchupServiceStatusCallback.getServiceStatus(ServiceStatus.java:277) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
           at org.apache.pinot.common.utils.ServiceStatus$MultipleCallbackServiceStatusCallback.getServiceStatus(ServiceStatus.java:179) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]

Posted by "0xShubhamSolanki (via GitHub)" <gi...@apache.org>.
0xShubhamSolanki commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3304473504

   I'm working on a fix for this issue using a thread-safe KafkaConsumer wrapper with ReentrantLock synchronization. 
   
   Plan:
   - Create `ThreadSafeKafkaConsumer` wrapper class
   - Update `KafkaStreamMetadataProvider` to use wrapper
   - Modify `RealtimeSegmentDataManager` for safe offset fetching
   
   Timeline: PR will be in next 3-4 days
   
   Will submit PR #16314 soon. Any specific requirements or concerns? @Jackie-Jiang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3616384772

   This is fixed in:
   https://github.com/apache/pinot/pull/17089


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #16314: Fix multi-threaded access to KafkaConsumer
URL: https://github.com/apache/pinot/issues/16314


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3331411448

   cc @KKcorps @noob-se7en to help with this effort


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org