You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/07/09 22:32:52 UTC
[I] Fix multi-threaded access to KafkaConsumer [pinot]
Jackie-Jiang opened a new issue, #16314:
URL: https://github.com/apache/pinot/issues/16314
`KafkaConsumer` cannot be used in a multi-threaded manner
```
2025/07/09 22:30:06.371 WARN [RealtimeSegmentDataManager_*] [Start a Pinot [SERVER]] Cannot fetch stream offset with criteria OffsetCriteria{_offsetType=LARGEST, _offsetString='largest'} for clientId * and partitionGroupId 16 with maxWaitTime 5000
java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2445) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2429) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:2213) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchStreamPartitionOffset(KafkaStreamMetadataProvider.java:103) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchStreamOffset(RealtimeSegmentDataManager.java:1778) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1753) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1758) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.server.starter.helix.FreshnessBasedConsumptionStatusChecker.isSegmentCaughtUp(FreshnessBasedConsumptionStatusChecker.java:76) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.server.starter.helix.IngestionBasedConsumptionStatusChecker.getNumConsumingSegmentsNotReachedIngestionCriteria(IngestionBasedConsumptionStatusChecker.java:96) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.common.utils.ServiceStatus$RealtimeConsumptionCatchupServiceStatusCallback.getServiceStatus(ServiceStatus.java:277) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at org.apache.pinot.common.utils.ServiceStatus$MultipleCallbackServiceStatusCallback.getServiceStatus(ServiceStatus.java:179) ~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]
Posted by "0xShubhamSolanki (via GitHub)" <gi...@apache.org>.
0xShubhamSolanki commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3304473504
I'm working on a fix for this issue using a thread-safe KafkaConsumer wrapper with ReentrantLock synchronization.
Plan:
- Create `ThreadSafeKafkaConsumer` wrapper class
- Update `KafkaStreamMetadataProvider` to use wrapper
- Modify `RealtimeSegmentDataManager` for safe offset fetching
Timeline: PR will be in next 3-4 days
Will submit PR #16314 soon. Any specific requirements or concerns? @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3616384772
This is fixed in:
https://github.com/apache/pinot/pull/17089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #16314: Fix multi-threaded access to KafkaConsumer
URL: https://github.com/apache/pinot/issues/16314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix multi-threaded access to KafkaConsumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16314:
URL: https://github.com/apache/pinot/issues/16314#issuecomment-3331411448
cc @KKcorps @noob-se7en to help with this effort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org