You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Asif Shahid <as...@gmail.com> on 2025/12/05 21:31:00 UTC

Clarification requested

In the transform API :
def transformUpWithPruning(cond: TreePatternBits => Boolean,
    ruleId: RuleId = UnknownRuleId)(rule: PartialFunction[BaseType,
BaseType])

my understanding is that the information whether a TreeNode ( be Plan or
Expression) keeps the state in a local var in the Plan, to track if the
rule is effective or not, based on first time traversal results.

Now if say a batch has two rules ( Rule A and rule B) , such that rule B
changes the node ( Plan or Expression) , but rule A does not..
Am I right in understanding that in such case the effect of pruning will be
lost for rule A, as rule B is creating new nodes and hence information will
be lost?

Regards
Asif

Re: Clarification requested

Posted by Asif Shahid <as...@gmail.com>.
Thank you for the clarification..
Right , the supposition makes sense, for generalized pruning behaviour, I
suppose that is all that can be done...


On Sun, Dec 7, 2025, 8:10 PM Wenchen Fan <cl...@gmail.com> wrote:

> You can find the initial commit in the commit history:
> https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631
>
> Yes, and it's reasonable as we can't know if the new node has the same
> pruning behavior of the previous node.
>
> On Sat, Dec 6, 2025 at 5:33 AM Asif Shahid <as...@gmail.com> wrote:
>
>> In the transform API :
>> def transformUpWithPruning(cond: TreePatternBits => Boolean,
>>     ruleId: RuleId = UnknownRuleId)(rule: PartialFunction[BaseType,
>> BaseType])
>>
>> my understanding is that the information whether a TreeNode ( be Plan or
>> Expression) keeps the state in a local var in the Plan, to track if the
>> rule is effective or not, based on first time traversal results.
>>
>> Now if say a batch has two rules ( Rule A and rule B) , such that rule B
>> changes the node ( Plan or Expression) , but rule A does not..
>> Am I right in understanding that in such case the effect of pruning will
>> be lost for rule A, as rule B is creating new nodes and hence information
>> will be lost?
>>
>> Regards
>> Asif
>>
>

Re: Clarification requested

Posted by Wenchen Fan <cl...@gmail.com>.
You can find the initial commit in the commit history:
https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631

Yes, and it's reasonable as we can't know if the new node has the same
pruning behavior of the previous node.

On Sat, Dec 6, 2025 at 5:33 AM Asif Shahid <as...@gmail.com> wrote:

> In the transform API :
> def transformUpWithPruning(cond: TreePatternBits => Boolean,
>     ruleId: RuleId = UnknownRuleId)(rule: PartialFunction[BaseType,
> BaseType])
>
> my understanding is that the information whether a TreeNode ( be Plan or
> Expression) keeps the state in a local var in the Plan, to track if the
> rule is effective or not, based on first time traversal results.
>
> Now if say a batch has two rules ( Rule A and rule B) , such that rule B
> changes the node ( Plan or Expression) , but rule A does not..
> Am I right in understanding that in such case the effect of pruning will
> be lost for rule A, as rule B is creating new nodes and hence information
> will be lost?
>
> Regards
> Asif
>