You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/03 11:31:54 UTC

[GitHub] [arrow-datafusion] alamb opened a new pull request, #3351: MINOR: Add comments about rewrite_disjunctive_predicate

alamb opened a new pull request, #3351:
URL: https://github.com/apache/arrow-datafusion/pull/3351

   # Which issue does this PR close?
   re https://github.com/apache/arrow-datafusion/issues/217 and https://github.com/apache/arrow-datafusion/issues/78
   
    # Rationale for this change
   While was working on https://github.com/apache/arrow-datafusion/pull/3334, @DhamoPS noticed that TPCH Q19 was not being rewritten, despite the code in https://github.com/apache/arrow-datafusion/blob/master/datafusion/optimizer/src/rewrite_disjunctive_predicate.rs from @xudong963 
   
   However, I think it was not very clear what the intent of the rewrite pass was so I figured that some docstrings would help
   
   # What changes are included in this PR?
   Add docstrings to optimizer pass, and move structure up so the comments are more discoverable
   
   # Are there any user-facing changes?
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] ursabot commented on pull request #3351: MINOR: Add comments about rewrite_disjunctive_predicate

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #3351:
URL: https://github.com/apache/arrow-datafusion/pull/3351#issuecomment-1236348335

   Benchmark runs are scheduled for baseline = 0bbf12740b3c825af48a3d69e07bc53a74dbefb3 and contender = d7fa064679bcfe14df092deb9e937b7b39b21c33. d7fa064679bcfe14df092deb9e937b7b39b21c33 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5f23092b31234f1a940ca31d8cd40191...c79e4174d3cc47c7892f30a217f7e8c0/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/b553115d26f841f494871d06d8326181...26c24be78f1141848d8cb3063b137bd1/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/76d6eb8a995e4f6e9fe14e2cce9fb673...a46c2130284d4277a69eef66981275ff/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8c7f9fce5e9b4ac7b6d2391d47304238...a51b46e1311a41398e6fc601535894f4/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #3351: MINOR: Add comments about rewrite_disjunctive_predicate

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3351:
URL: https://github.com/apache/arrow-datafusion/pull/3351#issuecomment-1236106816

   # [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3351](https://codecov.io/gh/apache/arrow-datafusion/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b519539) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/eb8c81f2ddacdc71086d98b96cdaa00233cb8749?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eb8c81f) will **increase** coverage by `0.00%`.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@           Coverage Diff           @@
   ##           master    #3351   +/-   ##
   =======================================
     Coverage   85.46%   85.46%           
   =======================================
     Files         294      294           
     Lines       54155    54155           
   =======================================
   + Hits        46284    46286    +2     
   + Misses       7871     7869    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ion/optimizer/src/rewrite\_disjunctive\_predicate.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3Jld3JpdGVfZGlzanVuY3RpdmVfcHJlZGljYXRlLnJz) | `96.75% <100.00%> (ø)` | |
   | [datafusion/common/src/scalar.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3NjYWxhci5ycw==) | `84.65% <0.00%> (+0.06%)` | :arrow_up: |
   | [datafusion/expr/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `77.61% <0.00%> (+0.16%)` | :arrow_up: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3351: MINOR: Add comments about rewrite_disjunctive_predicate

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3351:
URL: https://github.com/apache/arrow-datafusion/pull/3351#discussion_r962142247


##########
datafusion/optimizer/src/rewrite_disjunctive_predicate.rs:
##########
@@ -23,6 +23,152 @@ use datafusion_expr::Expr::BinaryExpr;
 use datafusion_expr::{Expr, LogicalPlan, Operator};
 use std::sync::Arc;
 
+/// Optimizer pass that rewrites predicates of the form
+///
+/// ```text
+/// (A = B AND <expr1>) OR (A = B AND <expr2>) OR ... (A = B AND <exprN>)
+/// ```
+///
+/// Into
+/// ```text
+/// (A = B) AND (<expr1> OR <expr2> OR ... <exprN> )
+/// ```
+///
+/// Predicates connected by `OR` typically not able to be broken down
+/// and distributed as well as those connected by `AND`.
+///
+/// The idea is to rewrite predicates into `good_predicate1 AND
+/// good_predicate2 AND ...` where `good_predicate` means the
+/// predicate has special support in the execution engine.
+///
+/// Equality join predicates (e.g. `col1 = col2`), or single column
+/// expressions (e.g. `col = 5`) are examples of predicates with
+/// special support.
+///
+/// # TPCH Q19
+///
+/// This optimization is admittedly somewhat of a niche usecase. It's
+/// main use is that it appears in TPCH Q19 and is required to avoid a
+/// CROSS JOIN.
+///
+/// Specificially, Q19 has a WHERE clause that looks like
+///
+/// ```sql
+/// where
+///   p_partkey = l_partkey
+///   and l_shipmode in (‘AIR’, ‘AIR REG’)
+///   and l_shipinstruct = ‘DELIVER IN PERSON’
+///   and (
+///     (
+///       and p_brand = ‘[BRAND1]’
+///       and p_container in ( ‘SM CASE’, ‘SM BOX’, ‘SM PACK’, ‘SM PKG’)
+///       and l_quantity >= [QUANTITY1] and l_quantity <= [QUANTITY1] + 10
+///       and p_size between 1 and 5
+///     )
+///     or
+///     (
+///       and p_brand = ‘[BRAND2]’
+///       and p_container in (‘MED BAG’, ‘MED BOX’, ‘MED PKG’, ‘MED PACK’)
+///       and l_quantity >= [QUANTITY2] and l_quantity <= [QUANTITY2] + 10
+///       and p_size between 1 and 10
+///     )
+///     or
+///     (
+///       and p_brand = ‘[BRAND3]’
+///       and p_container in ( ‘LG CASE’, ‘LG BOX’, ‘LG PACK’, ‘LG PKG’)
+///       and l_quantity >= [QUANTITY3] and l_quantity <= [QUANTITY3] + 10
+///       and p_size between 1 and 15
+///     )
+/// )
+/// ```
+///
+/// Niavely planning this query will result in a CROSS join with that
+/// single large OR filter. However, rewriting it using the rewrite in
+/// this pass results in a proper join predicate, `p_partkey = l_partkey`:
+///
+/// ```sql
+/// where
+///   p_partkey = l_partkey
+///   and l_shipmode in (‘AIR’, ‘AIR REG’)
+///   and l_shipinstruct = ‘DELIVER IN PERSON’
+///   and (
+///     (
+///       and p_brand = ‘[BRAND1]’
+///       and p_container in ( ‘SM CASE’, ‘SM BOX’, ‘SM PACK’, ‘SM PKG’)
+///       and l_quantity >= [QUANTITY1] and l_quantity <= [QUANTITY1] + 10
+///       and p_size between 1 and 5
+///     )
+///     or
+///     (
+///       and p_brand = ‘[BRAND2]’
+///       and p_container in (‘MED BAG’, ‘MED BOX’, ‘MED PKG’, ‘MED PACK’)
+///       and l_quantity >= [QUANTITY2] and l_quantity <= [QUANTITY2] + 10
+///       and p_size between 1 and 10
+///     )
+///     or
+///     (
+///       and p_brand = ‘[BRAND3]’
+///       and p_container in ( ‘LG CASE’, ‘LG BOX’, ‘LG PACK’, ‘LG PKG’)
+///       and l_quantity >= [QUANTITY3] and l_quantity <= [QUANTITY3] + 10
+///       and p_size between 1 and 15
+///     )
+/// )
+/// ```
+///
+#[derive(Default)]

Review Comment:
   I just moved this code to the top of the module so that the comments describing what it does can be attached to the structure as well as being at the top of the file, for better discoverability
   
   No change in behavior is intended



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #3351: MINOR: Add comments about rewrite_disjunctive_predicate

Posted by GitBox <gi...@apache.org>.
alamb merged PR #3351:
URL: https://github.com/apache/arrow-datafusion/pull/3351


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org