[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext

["Edoardo Vacchi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499542#comment-14499542 ] 

Edoardo Vacchi commented on SPARK-6320:
---------------------------------------

My use case is customizing strategies so that operations can be pruned/rewritten, depending on metadata in a custom catalog. For instance, consider the case of a DataFrame where a column X is known to be non-nullable; if the query contains the filter {{where('x is not null)}} the filter can be pruned with a rule of the form:

{code:java}
      case logical.Filter(IsNotNull(e: NamedExpression), child)
        if customCatalog.isNonNull(e) => {
          planLater(child) :: Nil
{code}

I wonder why is {{SparkPlanner}} nested inside {{SQLContext}}. Cass {{SparkPlanner}} may require the {{SQLContext}}, {{SQLConf}} as constructor parameters and be defined in its own class (and the same could be said for QueryExecution).  suggestion would be to refactor out {{SparkPlanner}}. Advanced users would then be able to extend it like any other class.

> Adding new query plan strategy to SQLContext
> --------------------------------------------
>
>                 Key: SPARK-6320
>                 URL: https://issues.apache.org/jira/browse/SPARK-6320
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>            Reporter: Youssef Hatem
>            Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a new class which extends {{Strategy}}. In my new class I need to call {{planLater}} function. However this method is defined in {{SparkPlanner}} (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new strategy is to define my strategy inside another class that extends {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will have to extend the {{SQLContext}} such that I can override the {{planner}} field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to require extending {{SQLContext}} (unless I am doing it wrong and there is a better way to do it).
> Thanks a lot,
> Youssef



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org