You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "924060929 (via GitHub)" <gi...@apache.org> on 2023/02/24 12:55:38 UTC

[GitHub] [doris] 924060929 opened a new pull request, #17126: [refactor](Nereids) Refactor rewrite framework

924060929 opened a new pull request, #17126:
URL: https://github.com/apache/doris/pull/17126

   # Proposed changes
   This pr refactor the rewrite framework from memo to plan tree, and speed up the analyze/rewrite stage.
   
   Changes:
   - abandoned memo in the analysis/rewrite stage, so that we can skip some action , like new GroupExpression, distinct GroupExpression in the memo(high cost), update children to GroupPlan
   - change the most of rule to static rule, so that we can skip initialize Analyzer/Rewriter and lots of rule in every query. but some rules need context, like visitor rule, create rule at the runtime make it is easy to use, so make `custom` rule can help us to create it.
   - remove the `logger` field in the Job, Job are generated in large quantities at runtime, we don't need to use logger so save huge time to init logger.
   - skip some rule as far as possible, e.g. `SelectMaterializedIndexWithoutAggregate`, skip select mv if the table not exist rullup.
   - add some cache for frequent operation, like get Job.getDisableRule, Plan.getUnboundExpression
   - new bottom up rewrite rule, it can keep traverse multiple new plan which return by rules, this feature depends on `Plan.mutableState`, it is necessary to add this variable field for plan. if the plan is fully immutable, we must use withXxx to renew the plan and set the state for it, this not only take more runtime overhead and developing workload. another reason is we need multiple mutable state, e.g. whether is applied the rule,  whether this plan is manage by the rewrite framework. the good side of mutable state is the efficient, but I suggest we don't direct use mutable state in the rule as far as possible, if we need use it, please wrap the mutable state in the framework to correct modification and release it, a good example is `AppliedAwareRuleCondition`, it can update and get the state: whether this plan is applied to a rule before.
   - refactor the `EliminateUnnecessaryProject` by CustomRewritor, fix the problem which eliminate some Project which decided the query output order, the case is limit(project), sort(project).
   
   
   
   ## Checklist(Required)
   
   * [ ] Does it affect the original behavior
   * [x] Has unit tests been added
   * [ ] Has document been added or modified
   * [ ] Does it need to update dependencies
   * [ ] Is this PR support rollback (If NO, please explain WHY)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17126:
URL: https://github.com/apache/doris/pull/17126#issuecomment-1447727080

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] 924060929 commented on pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #17126:
URL: https://github.com/apache/doris/pull/17126#issuecomment-1443647666

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] 924060929 commented on pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #17126:
URL: https://github.com/apache/doris/pull/17126#issuecomment-1445698257

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morrySnow merged pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #17126:
URL: https://github.com/apache/doris/pull/17126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17126:
URL: https://github.com/apache/doris/pull/17126#issuecomment-1447727120

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] 924060929 commented on pull request #17126: [refactor](Nereids) Refactor rewrite framework

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #17126:
URL: https://github.com/apache/doris/pull/17126#issuecomment-1445782960

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org