You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2020/04/12 12:53:45 UTC
InferFiltersFromConstraints logical optimization rule and Optimizer.defaultBatches?
Hi,
I'm curious why there is a need to include InferFiltersFromConstraints
logical optimization in operatorOptimizationRuleSet value [1] that seems to
be only to exclude it later [2]?
In other words, I think that simply removing InferFiltersFromConstraints
from operatorOptimizationRuleSet value [1] would make no change (except
removing a "dead code").
Does this make sense? Could I be missing something?
[1]
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L80
[2]
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L115
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
Re: InferFiltersFromConstraints logical optimization rule and Optimizer.defaultBatches?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Jungtaek,
Thanks a lot for your answer. What you're saying reflects my understanding
perfectly. There's a small change, but makes understanding where rules are
used much simpler (= less confusing). I'll propose a PR and see where it
goes from there. Thanks!
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
On Wed, Apr 15, 2020 at 7:55 AM Jungtaek Lim <ka...@gmail.com>
wrote:
> Please correct me if I'm missing something. At a glance, your statements
> look correct if I understand correctly. I guess it might be simply missed,
> but it sounds as pretty trivial one as only a line can be removed safely
> which won't affect anything. (filterNot should be retained even we remove
> the line to prevent extended rules to break this)
>
> On Sun, Apr 12, 2020 at 9:54 PM Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> I'm curious why there is a need to include InferFiltersFromConstraints
>> logical optimization in operatorOptimizationRuleSet value [1] that seems to
>> be only to exclude it later [2]?
>>
>> In other words, I think that simply removing InferFiltersFromConstraints
>> from operatorOptimizationRuleSet value [1] would make no change (except
>> removing a "dead code").
>>
>> Does this make sense? Could I be missing something?
>>
>> [1]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L80
>> [2]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L115
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books <https://books.japila.pl/>
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> <https://twitter.com/jaceklaskowski>
>>
>
Re: InferFiltersFromConstraints logical optimization rule and Optimizer.defaultBatches?
Posted by Jungtaek Lim <ka...@gmail.com>.
Please correct me if I'm missing something. At a glance, your statements
look correct if I understand correctly. I guess it might be simply missed,
but it sounds as pretty trivial one as only a line can be removed safely
which won't affect anything. (filterNot should be retained even we remove
the line to prevent extended rules to break this)
On Sun, Apr 12, 2020 at 9:54 PM Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> I'm curious why there is a need to include InferFiltersFromConstraints
> logical optimization in operatorOptimizationRuleSet value [1] that seems to
> be only to exclude it later [2]?
>
> In other words, I think that simply removing InferFiltersFromConstraints
> from operatorOptimizationRuleSet value [1] would make no change (except
> removing a "dead code").
>
> Does this make sense? Could I be missing something?
>
> [1]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L80
> [2]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L115
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>