[jira] [Created] (SPARK-28217) Allow a custom statistics logical plan visitor to be plugged in.

["Terry Kim (JIRA)" <ji...@apache.org>]

Terry Kim created SPARK-28217:
---------------------------------

             Summary: Allow a custom statistics logical plan visitor to be plugged in.
                 Key: SPARK-28217
                 URL: https://issues.apache.org/jira/browse/SPARK-28217
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Terry Kim


Spark currently has two built-in statistics plan visitor: SizeInBytesOnlyStatsPlanVisitor and BasicStatsPlanVisitor. However, this is a bit limited since there is no way to plug in a custom plan visitor - from which a custom query optimizer can benefit from.

We can provide a Spark conf that the user can specify to override the built-in plan visitor:

{code:scala}
// First create your custom stat plan visitor.
class MyStatsPlanVisitor extends LogicalPlanVisitor[Statistics] {
  // Implement LogicalPlanVisitor[Statistics] trait
}

// Set the visitor via Spark conf.
spark.conf.set("spark.sql.catalyst.statsPlanVisitorClass", "MyStatsPlanVisitor")

// Now, stat() on a LogicalPlan object will use MyStatsPlanVisitor as a stat plan visitor.
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org