You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Murphy <ha...@gmail.com> on 2015/01/23 00:19:43 UTC

query planner design doc?

Hi-

Quick question: is there a design doc (or something more than “look at the code”) for the query planner for Spark SQL (i.e., the component that takes…Catalyst?…operator trees and translates them into SPARK operations)?

Thanks,
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: query planner design doc?

Posted by Michael Armbrust <mi...@databricks.com>.
No, are you looking for something in particular?

On Fri, Jan 23, 2015 at 9:44 AM, Nicholas Murphy <ha...@gmail.com>
wrote:

> Okay, thanks.  The design document mostly details the infrastructure for
> optimization strategies but doesn’t detail the strategies themselves.  I
> take it the set of strategies are basically embodied in
> SparkStrategies.scala...is there a design doc/roadmap/JIRA issue detailing
> what strategies exist and which are planned?
>
> Thanks,
> Nick
>
> On Jan 22, 2015, at 7:45 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
> Here is the initial design document for catalyst :
>
> https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit
>
> Strategies (many of which are in SparkStragegies.scala) are the part that
> creates the physical operators from a catalyst logical plan.  These
> operators have execute() methods that actually call RDD operations.
>
> On Thu, Jan 22, 2015 at 3:19 PM, Nicholas Murphy <ha...@gmail.com>
> wrote:
>
>> Hi-
>>
>> Quick question: is there a design doc (or something more than “look at
>> the code”) for the query planner for Spark SQL (i.e., the component that
>> takes…Catalyst?…operator trees and translates them into SPARK operations)?
>>
>> Thanks,
>> Nick
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>

Re: query planner design doc?

Posted by Nicholas Murphy <ha...@gmail.com>.
Okay, thanks.  The design document mostly details the infrastructure for optimization strategies but doesn’t detail the strategies themselves.  I take it the set of strategies are basically embodied in SparkStrategies.scala...is there a design doc/roadmap/JIRA issue detailing what strategies exist and which are planned?

Thanks,
Nick

> On Jan 22, 2015, at 7:45 PM, Michael Armbrust <mi...@databricks.com> wrote:
> 
> Here is the initial design document for catalyst :
> https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit <https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit>
> 
> Strategies (many of which are in SparkStragegies.scala) are the part that creates the physical operators from a catalyst logical plan.  These operators have execute() methods that actually call RDD operations.
> 
> On Thu, Jan 22, 2015 at 3:19 PM, Nicholas Murphy <halcyonic@gmail.com <ma...@gmail.com>> wrote:
> Hi-
> 
> Quick question: is there a design doc (or something more than “look at the code”) for the query planner for Spark SQL (i.e., the component that takes…Catalyst?…operator trees and translates them into SPARK operations)?
> 
> Thanks,
> Nick
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> For additional commands, e-mail: dev-help@spark.apache.org <ma...@spark.apache.org>
> 
>

Re: query planner design doc?

Posted by Michael Armbrust <mi...@databricks.com>.
Here is the initial design document for catalyst :
https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit

Strategies (many of which are in SparkStragegies.scala) are the part that
creates the physical operators from a catalyst logical plan.  These
operators have execute() methods that actually call RDD operations.

On Thu, Jan 22, 2015 at 3:19 PM, Nicholas Murphy <ha...@gmail.com>
wrote:

> Hi-
>
> Quick question: is there a design doc (or something more than “look at the
> code”) for the query planner for Spark SQL (i.e., the component that
> takes…Catalyst?…operator trees and translates them into SPARK operations)?
>
> Thanks,
> Nick
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>