You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jadami10 (via GitHub)" <gi...@apache.org> on 2023/09/01 20:36:00 UTC

[GitHub] [pinot] jadami10 opened a new issue, #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

jadami10 opened a new issue, #11488:
URL: https://github.com/apache/pinot/issues/11488

   We caught this as we're trying to use the new ingestion lag metric. There is a [5 minute delay](https://github.com/apache/pinot/blob/2f48060d37f94a4570a0b15b13a58391e177ee8a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java#L109C41-L109C41) before the lead controller resource rebalances partitions. When we unexpectedly lose a controller, maybe this helps. But Pinot has a path for gracefully shutting down a controller, and this shouldn't impact ingestion lag.
   
   Potential solutions here:
   - interestingly enough, helix has added [on demand rebalance](https://github.com/apache/helix/pull/2601) support as of last week. I know we recently upgraded to helix 1.x, so maybe a further upgrade is possible
   - we can make the rebalance delay configurable. I think for us, we're just going to test with 0, or no delay. It doesn't seem like rebalancing is particularly expensive, and server retry segment commits quite frequently. This raises a secondary issue that the `leadControllerResource` cannot be update once created, though that's a simple enough fix.
   
   Happy to hear other thoughts here or maybe even a background on if 5 minutes was choses for a reason.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Losing a controller leads to a minimum 5 minute window where segments cannot be committed [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-5824548336

   This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1723396128

   Those are all valid points. For us, a lot of it is mitigated because we are overprovisioned on the controller side. Every cluster has 3 controllers, I don't see any GC taking > 1 ms, all segment uploads are done directly to deep store, and metrics being temporarily wrong is ok. But the ingestion lag issue was persistent, happening multiple times a day.
   
   But we'd ideally use helix's new, on-demand rebalance feature once helix is updated far enough in this repo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1703331160

   @jackjlli Can you share more context on why there is a 5 minutes delay? I can see we want to avoid switching leader, but what is the side effect of switching leader?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1703312163

   This also ties into another async operation where servers looking for the lead controller cache that result for 30 seconds. So this would need to play well with that, too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] kishoreg commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "kishoreg (via GitHub)" <gi...@apache.org>.
kishoreg commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1723570737

   +1 to what Johan is asking for. We need to fix the design/impl. 
   
   Subbu, flapping can be detected easily. In fact, helix will put the participant in disabled state if it flaps 5 times in a minute. We cannot have the happy path impacted to handle edge case.
   
   for restart, we should be able to handle this gracefully.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1714088033

   thanks for the comment jack. I think our issue ended up being a combination of things:
   - our clusters are large enough (in number of tables + segments) where there's effectively always some segment sealing
   - we want to keep ingestion lag < 30 seconds
   - our controllers are replaced every 7 days
   - replacement is not fast because we take down the existing controller before bringing up the new one
   
   But configuring the rebalance to be 5 seconds internally has been working well for us


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Losing a controller leads to a minimum 5 minute window where segments cannot be committed [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed
URL: https://github.com/apache/pinot/issues/11488


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jackjlli commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1713288970

   The purpose of introducing the 5 min delay is that most of the time controllers encounter restart/deployment other than host replacement. And most of the time Pinot controller can be fully started up within 5 mins, that's why 5 min is initially configured. In addition, setting this delay can help that most of the message logs for particular tables remain in the consistent controllers.
   
   According to the discussion in this [issue](https://github.com/apache/helix/issues/2594), on-demand rebalance is not in the current used Helix version (neither 1.0.4 or 1.3.0) in Pinot yet. I'm fine with making the delay configurable to meet your needs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] mcvsubbu commented on issue #11488: Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-1722641421

   Consider what will happen when the lead controller keeps switching back and forth for tables. We run periodic jobs for the table on the lead controller and these jobs emit metrics. Also, if you set the time too short, and one of the controllers undergoes  a slightly long GC, you are in danger of the controller losing mastership (as per helix) but continuing the last operation it was doing. This could lead to inconsistencies in the system.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Losing a controller leads to a minimum 5 minute window where segments cannot be committed [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11488:
URL: https://github.com/apache/pinot/issues/11488#issuecomment-6405467901

   This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org