You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jadami10 (via GitHub)" <gi...@apache.org> on 2023/10/16 15:46:07 UTC
[I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
jadami10 opened a new issue, #11812:
URL: https://github.com/apache/pinot/issues/11812
I believe this issue originates here, https://github.com/apache/pinot/blob/a5b3483a838839e606b251f2bb6e0d0b22f75a93/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L533. When committing segments, the controller also sets up the new segments. It requests the new partition group metadata to see if there's any new partitions that need to be consumed from.
This eventually calls `computePartitionGroupMetadata` which also fetches the offsets for all partitions. In our case, we have thousands of of partitions, and sometimes this call just fails because kafka times out or returns an error. In our case, we see ~10-20 minutes of ingestion lag until the segment commit is retried.
The code itself doesn't even need the offsets, it just needs the partition count and the partition group ids. But it would be great if someone could double check me there.
Initially I can think of 2 backwards compatible solutions:
- use the existing `fetchPartitionCount` to get the partition count. add a new method to `StreamMetadataProvider` to fetch just the partitionIds. It can default to using `computePartitionGroupMetadata`, and we can override it in our own class
- add a new `computePartitionGroupMetadata` function that takes a `withOffsets` boolean to determine the behavior.
I'm leaning towards the former since it's a more explicit solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #11812: Unnecessary offset request in segment commit flow causes instability and ingestion lag
URL: https://github.com/apache/pinot/issues/11812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1773552130
@priyen It depends. You may check the time spent during regular segment commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1773315092
@Jackie-Jiang, I've confirmed that every force commit causes an offset request from Pinot->Kafka. For a table with hundreds of consuming segments, this caused hundreds of offset requests.
Another potential issue I see is this table experienced ~2 minutes of ingestion lag between requesting the force commit and the force commit completing. I haven't looked at the code yet, but should we do these commits in batches? Meaning, should we only tell a segment to commit once a previous one has completed to avoid lag while waiting for other work to complete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1792809555
> Another potential issue I see is this table experienced ~2 minutes of ingestion lag between requesting the force commit and the force commit completing
This consumption lag is table wide after force-commit is issued. https://github.com/apache/pinot/issues/11950 proposes to stagger committing different partitions over longer period of time so that you can control how you want to experience the consumption lag: all at once (current behavior) or spread over longer period of time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1764775181
cc @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1769646769
> Curious on how you find this hotspot? Do you see this method showing up in the profile?
Didn't need a profile. This times out after 25 seconds for us and causes ingestion to pause. I traced it back from the stack trace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1773410261
> Another potential issue I see is this table experienced ~2 minutes of ingestion lag between requesting the force commit and the force commit completing. I haven't looked at the code yet, but should we do these commits in batches? Meaning, should we only tell a segment to commit once a previous one has completed to avoid lag while waiting for other work to complete?
After requesting the force commit, server needs to seal the current consuming segment and commit it, which can explain the ~2 minutes lag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "priyen (via GitHub)" <gi...@apache.org>.
priyen commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1764909174
This may be contributing to the slow force commit behaviour we noticed and end up filing tickets for. We did note the calls to get offset info, but didnt end up looking further among the other things we tried to look into (table level locking in the commit protocol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1769581710
Good catch! I believe we should be able to refactor the code and only read the partition count.
Curious on how you find this hotspot? Do you see this method showing up in the profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Unnecessary offset request in segment commit flow causes instability and ingestion lag [pinot]
Posted by "priyen (via GitHub)" <gi...@apache.org>.
priyen commented on issue #11812:
URL: https://github.com/apache/pinot/issues/11812#issuecomment-1773411439
Is sealing expected to take that long?
On Fri, Oct 20, 2023, 5:21 p.m. Xiaotian (Jackie) Jiang <
***@***.***> wrote:
> Another potential issue I see is this table experienced ~2 minutes of
> ingestion lag between requesting the force commit and the force commit
> completing. I haven't looked at the code yet, but should we do these
> commits in batches? Meaning, should we only tell a segment to commit once a
> previous one has completed to avoid lag while waiting for other work to
> complete?
>
> After requesting the force commit, server needs to seal the current
> consuming segment and commit it, which can explain the ~2 minutes lag.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pinot/issues/11812#issuecomment-1773410261>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACOCX4ABVQCRGZ3EYD2LZRTYALTNPAVCNFSM6AAAAAA6CNGIGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTGQYTAMRWGE>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org