You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/24 06:36:04 UTC
[GitHub] [spark] ulysses-you commented on issue #26875: [SPARK-30245][SQL]
Add cache for Like and RLike when pattern is not static
ulysses-you commented on issue #26875: [SPARK-30245][SQL] Add cache for Like and RLike when pattern is not static
URL: https://github.com/apache/spark/pull/26875#issuecomment-568670046
Yes, it just has a few scene and it is disencouraging. But when it happens, spark will run very slowly.
I meet a scene that check two table which storage some file locations, then use join and like to compare location. There is only one way spark can choose is `BroadcastNestedLoopJoin`.
E.G. table t1 and table t2 both have 1 million rows, and t1 has 200 partitions, t2 is broadcasted, then spark will compile pattern 1 million * 1 million count. But with cache, just compile 1 million * 200 count.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org