You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/04 13:44:24 UTC
[GitHub] [pinot] jadami10 opened a new issue, #9014: Allowing empty segments for all stream consumers
jadami10 opened a new issue, #9014:
URL: https://github.com/apache/pinot/issues/9014
In this part of the code, https://github.com/apache/pinot/blob/2b82366c369cf668175695605f0b77d7105f4aff/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L301-L304, we extend the segment build time by 1 hour if no rows were consumed. We have some common cases where certain partitions may never have data. We are either filtering out a lot of the data or there's a temporary error in our ingestion plugin. We also use a custom ingestion plugin.
But by forever extending the lease like this, whenever we restart the server, the server attempts to consume going back days or weeks and causes high CPU usage and throttling for both the server and kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #9014: Allowing empty segments for all stream consumers
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1174620801
Hi, what I am not able to understand is, why is server consuming days/weeks back data? The consumer only deals with the offsets. It just starts consuming after the last offset that was consumed. If there was no data published, then it shouldn't be an issue imo. The rest of the partitions which had data should have already committed and so they shouldn't be getting consumed from start as well.
Can you add you table config here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #9014: Allowing empty segments for all stream consumers
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175211935
This seems to be a valid use case, and we should be able to support empty segment commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #9014: Allowing empty segments for all stream consumers
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175446453
@jadami10 Yeah, in your scenario it makes sense. Yep, we can take it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #9014: Allowing empty segments for all stream consumers
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9014: Allowing empty segments for all stream consumers
URL: https://github.com/apache/pinot/issues/9014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #9014: Allowing empty segments for all stream consumers
Posted by GitBox <gi...@apache.org>.
jadami10 commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175179709
The problem is that there is data published, and it's quite high volume. We are just filtering it all out in our decoder. So in this case, the server never actually commits the segment to advance the offsets. So a restart will cause it to go all the way back to the last time it saw a message it didn't filter out which can be hours or days.
Our table config is has a bunch of custom keys for our internal stream ingestion plugin, so i'm not sure it would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org