You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/02/28 18:30:17 UTC
[GitHub] [pinot] walterddr opened a new issue #8260: Multi-stage execution query engine
walterddr opened a new issue #8260:
URL: https://github.com/apache/pinot/issues/8260
We have come up with a design doc to support complex multi-stage query execution.
Motivation
===
Today's Pinot query engine only supports a single scatter-gather model. This design runs very well if most of the heavy-lifting computation can be push down to segment servers, however it also have some limitations we would like to address. Namely,
* Remove bottleneck in Pinot Broker when heavy reduce is needed.
* Support additional SQL features such as distributed join.
* Support standardized full SQL semantics
Proposal
===
We propose to build a new multi-stage query engine to address these, specifically
* a new SQL planner that can support full-SQL semantic planning.
* a new execution engine that can support multi-stage, DAG-based execution.
Design Doc
===
Please see https://docs.google.com/document/d/10-vL_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit# for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kishoreg commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1059886886
The most awaited feature!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1064396466
Lets knock this down. +1
With all the +1's above, and no -1, I think it is safe to conclude that we can move ahead with detailed design+impl phase @walterddr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1070855641
@walterddr can you please open up access to the design doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1064394083
+1 on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia edited a comment on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
siddharthteotia edited a comment on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1081598178
First PR for query planner merged to feature branch https://github.com/apache/pinot/pull/8340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1070855641
@walterddr can you please open up access to the design doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1064414242
Linking related old issue for reference - https://github.com/apache/pinot/issues/5664
This one should be used and the 5664 can remain open or be closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1081598178
First PR for query planner PR merged to feature branch https://github.com/apache/pinot/pull/8340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on issue #8260: Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1057284503
This is so exciting!
Strong +1 to this proposal. It'll shape Pinot into the next-generation architecture and unlock many additional capabilities. For example, from the Uber side, this could help us extend the current lookup join and explore other join strategies such as broadcast join and colocated join.
It'll be great if you can push your prototype to a feature branch, so we can review and contribute over that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on issue #8260: Add support for JOIN - Multi-stage execution query engine
Posted by GitBox <gi...@apache.org>.
xiangfu0 commented on issue #8260:
URL: https://github.com/apache/pinot/issues/8260#issuecomment-1064394783
+1000！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org