You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/20 01:32:08 UTC

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29807: [WIP][SPARK-32939][SQL]Avoid re-compute expensive expression

AngersZhuuuu commented on a change in pull request #29807:
URL: https://github.com/apache/spark/pull/29807#discussion_r491639643



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanner.scala
##########
@@ -31,6 +31,12 @@ class SparkPlanner(
     val experimentalMethods: ExperimentalMethods)
   extends SparkStrategies {
 
+  private val EXPENSIVE_EXPR_PREFIX = "expensive_col_"
+
+  def extractExpensiveExprs(e: Expression): Seq[Expression] = e.collect {
+    case gjo: GetJsonObject => gjo

Review comment:
       > If you handle only `GetJsonObject`, please narrow down the PR title exactly.
   
   Sorry for forgot to add tag `[WIP]`. Since I am not familiar with all expensive expr or function. 
   In our env,  many data stored as Json, we use this a lot and always expr with `get_json_object` is slow.
   
   So I made this and hope for some advise and to see if it is reasonable enough




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org