You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/04/05 19:57:58 UTC
[GitHub] [pinot] walterddr opened a new issue, #10555: [multistage] perf regression on ORDER BY queries
walterddr opened a new issue, #10555:
URL: https://github.com/apache/pinot/issues/10555
Example
===
see:
https://github.com/apache/pinot/actions/runs/4440707776/jobs/7794852115 vs https://github.com/apache/pinot/actions/runs/4441822565/jobs/7797366601
Research
===
seems like the issue occurs on https://github.com/apache/pinot/pull/10408
- Did a bit of research with @somandal and it seems like the issue was with the seenMail trigger is being swallowed due to priority queue insertion taking much longer than simply returning the block back.
- after modifying `DEFAULT_SCHEDULER_RELEASE_TIMEOUT_MS = 1_000; // 10_000;` for the RoundRobinScheduler it is becoming much faster.
Action items
===
- tagging @ankitsultana and @somandal and see what would be the best solution
- tagging @walterddr to come up with a regression test instead of having to figure it out by looking at the integration test log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1500574678
Yup sounds good. Closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498058419
Hmm I see.. this is also related to some other known gaps with how the callbacks are handled.
I haven't gotten a chance to work on it lately: https://github.com/apache/pinot/issues/10424
I think I can prioritize it for next week (we are also planning to finalize our prod build internally so this is aligned with that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr closed issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr closed issue #10555: [multistage] perf regression on ORDER BY queries
URL: https://github.com/apache/pinot/issues/10555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498059261
Some logs for the testing done to identify the RC:
One example Send <-> Receive interaction:
```
Sending data block: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}, split: 0
Sending data block: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}, split: 1
sending EOS mailbox id: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}
...
Time to add 7182 elements to priority queue: 2, time since start of operator: 513, mailbox: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}
openmailbox count: 1, eosMailboxCount: 0, returning block type: noop
Time to add 2818 elements to priority queue: 1, time since start of operator: 519, mailbox: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}
openmailbox count: 1, eosMailboxCount: 0, returning block type: noop
EOS received after since operator creation: (time to receive: 16545) mailbox: {"jobId":"4_2","from":"0@localhost:8422","to":"0@localhost:8422","senderStageId":2,"receiverStageId":1}
Sorted block construction took: 7, total time to sort data: 16552, openmailbox: 1, eosmailboxcount: 1
openmailbox count: 0, eosMailboxCount: 0, returning block type: eos
```
@ankitsultana let's discuss a potential solution for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1500543957
The fix for the ORDER BY regression is merged.
Still need some follow up on the improvements around callback handling cc @walterddr @ankitsultana -> If this is tracked in another issue perhaps we can close this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498530597
Took a look and I think this is what's happening:
Say we are sorting in MailboxReceiveOperator and there's a single receiving mailbox. Say these blocks were sent in order: `B1, B2, B3, EOS`.
Either via callback or during first run of the OpChain, we will consume `B1` and add it to the priority queue. However, even though we consumed something, we return a NoOp block to the top.
Since we receive a NoOp block, we will try to yield, in where we'll check whether there's any callback for this OpChain (by looking up `_seenMail`).
Say there is, in that case we will schedule the OpChain again and clear the `_seenMail` entry. In the second run we'll consume `B2` and try to yield again. This time there's no callback entry so stay yielded and wait for automatic promotion to happen.
A possible fix for this would be to add a new MailboxReceiveOperator, say `SortedMailboxReceiveOperator` which does something similar to `buildBroadcastHashTable` in the join operator, i.e. continuously poll mailboxes until you get null or EOS.
There's a separate issue of missed callbacks btw which I have been punting for a while. I'll invest more time on this ticket next week: https://github.com/apache/pinot/issues/10424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1499154762
Thanks for the analysis, I agree that for the sorting logic we need to keep polling until EOS or null is seen. I’ll take up this fix today. I don’t want to add a SortOperator on top as in the future we want to things like partitioned sort for window functions where we only sort within a partition instead of a full sort. Also the sort exchange means that any receiving operators above it can expect sorted data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498068543
Sounds good. Let me take an initial look and get back. We can sync up later this week on the follow-ups. Does that work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498070078
Yup that works. Let me know if you need any more information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498072889
One more thing I tried: I tried returning a No-op block right after adding to the `priorityQueue`, but still ran into the same issue. So there is some delay caused by adding to the `priorityQueue`, but it shouldn't result in so much slowness (and after this step is performed, the `SortOperator` should only be applying LIMIT and FETCH so the overall ORDER BY should still be reasonable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498067157
@ankitsultana I've been using `MultiStageEngineIntegrationTest::testGeneratedQueries()`
I modified some code to only run the `ORDER BY` queries as the others finish quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10555: [multistage] perf regression on ORDER BY queries
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10555:
URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498064127
Is there a way to repro this on local easily?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org