You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/04 13:44:24 UTC

[GitHub] [pinot] jadami10 opened a new issue, #9014: Allowing empty segments for all stream consumers

jadami10 opened a new issue, #9014:
URL: https://github.com/apache/pinot/issues/9014

   In this part of the code, https://github.com/apache/pinot/blob/2b82366c369cf668175695605f0b77d7105f4aff/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L301-L304, we extend the segment build time by 1 hour if no rows were consumed. We have some common cases where certain partitions may never have data. We are either filtering out a lot of the data or there's a temporary error in our ingestion plugin. We also use a custom ingestion plugin.
   
   But by forever extending the lease like this, whenever we restart the server, the server attempts to consume going back days or weeks and causes high CPU usage and throttling for both the server and kafka.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] KKcorps commented on issue #9014: Allowing empty segments for all stream consumers

Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1174620801

   Hi, what I am not able to understand is, why is server consuming days/weeks back data? The consumer only deals with the offsets. It just starts consuming after the last offset that was consumed. If there was no data published, then it shouldn't be an issue imo. The rest of the partitions which had data should have already committed and so they shouldn't be getting consumed from start as well.
   Can you add you table config here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] mcvsubbu commented on issue #9014: Allowing empty segments for all stream consumers

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175211935

   This seems to be a valid use case, and we should be able to support empty segment commits.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] KKcorps commented on issue #9014: Allowing empty segments for all stream consumers

Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175446453

   @jadami10 Yeah, in your scenario it makes sense. Yep, we can take it up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang closed issue #9014: Allowing empty segments for all stream consumers

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9014: Allowing empty segments for all stream consumers
URL: https://github.com/apache/pinot/issues/9014


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on issue #9014: Allowing empty segments for all stream consumers

Posted by GitBox <gi...@apache.org>.
jadami10 commented on issue #9014:
URL: https://github.com/apache/pinot/issues/9014#issuecomment-1175179709

   The problem is that there is data published, and it's quite high volume. We are just filtering it all out in our decoder. So in this case, the server never actually commits the segment to advance the offsets. So a restart will cause it to go all the way back to the last time it saw a message it didn't filter out which can be hours or days.
   
   Our table config is has a bunch of custom keys for our internal stream ingestion plugin, so i'm not sure it would be helpful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org