You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "anandkrshaw (via GitHub)" <gi...@apache.org> on 2024/07/31 04:32:49 UTC
[I] Pinot servers stop consuming even if there is a issue with a single malformed message [pinot]
anandkrshaw opened a new issue, #13722:
URL: https://github.com/apache/pinot/issues/13722
When I check the server logs this is the only error I see - I assume there should be an handling for malformed messages.
And pinot server should move ahead and not stop the consumption at all.
Is this expected behavior then let me know I will have pre-validation of the messages.
2024/07/30 17:49:06.260 ERROR [RealtimeSegmentDataManager_orders__2__0__20240730T1148Z] [orders__2__0__20240730T1148Z] Could not build segment
java.lang.IllegalArgumentException: Invalid format: "06-May-2024 08:45:30" is malformed at "May-2024 08:45:30"
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:186) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:834) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.writeMetadata(SegmentColumnarIndexCreator.java:506) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:463) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:391) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.buildByColumn(SegmentIndexCreationDriverImpl.java:363) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:138) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentInternal(RealtimeSegmentDataManager.java:1023) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentForCommit(RealtimeSegmentDataManager.java:900) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:811) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e58d74e3e9b380c93f12fc15cbc260866c0e1f78]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024/07/30 17:49:06.280 ERROR [RealtimeSegmentDataManager_orders__2__0__20240730T1148Z] [orders__2__0__20240730T1148Z] Could not build segment for orders__2__0__20240730T1148Z
<img width="714" alt="image" src="https://github.com/user-attachments/assets/6fc27893-d947-4d5a-a25c-58897b1ece17">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Pinot servers stop consuming even if there is a issue with a single malformed message [pinot]
Posted by "anandkrshaw (via GitHub)" <gi...@apache.org>.
anandkrshaw closed issue #13722: Pinot servers stop consuming even if there is a issue with a single malformed message
URL: https://github.com/apache/pinot/issues/13722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Pinot servers stop consuming even if there is a issue with a single malformed message [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #13722:
URL: https://github.com/apache/pinot/issues/13722#issuecomment-2259838788
There is a config to continue ingestion even if there's an error.
Ref: https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion#monitor-real-time-ingestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Pinot servers stop consuming even if there is a issue with a single malformed message [pinot]
Posted by "anandkrshaw (via GitHub)" <gi...@apache.org>.
anandkrshaw commented on issue #13722:
URL: https://github.com/apache/pinot/issues/13722#issuecomment-2259847194
Yes @tibrewalpratik17 , Got to know from Mayank
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org