[I] Avoid inferring full anti-join coverage from range bounds [datafusion]

["gabotechs (via GitHub)" <gi...@apache.org>]

gabotechs opened a new issue, #25617:
URL: https://github.com/apache/datafusion/issues/25617

   ## Describe the bug
   
   An anti join estimates almost complete customer coverage from customer-key range
   statistics in orders, despite a third of customers having no orders. Missing NDVs are
   replaced by range bounds and treated as evidence of matching values.
   
   ## To Reproduce
   
   From the repository root, with the CLI fix from
   [PR #25570](https://github.com/apache/datafusion/pull/25570) applied:
   
   ```sh
   cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
   cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
   repro_dir=$(mktemp -d)
   tpchgen-cli --scale-factor 1 --format parquet \
     --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
   cat > "$repro_dir/repro.sql" <<'SQL'
   SET datafusion.execution.target_partitions = 1;
   SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
   SELECT c_custkey FROM customer WHERE NOT EXISTS (SELECT 1 FROM orders WHERE o_custkey = c_custkey);
   SQL
   target/ci/dfbench statistics \
     --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
   ```
   
   Observed with `tpchgen-cli` 1.1.1 at
   [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67).
   Inspect the SELECT reports; ignore the empty `SET` reports.
   
   | Operator            | Node | Estimated rows | Actual rows |
   | ------------------- | ---- | -------------: | ----------: |
   | HashJoinExec (anti) | `0`  |              1 |      50,004 |
   
   ## Expected behavior
   
   Distinguish NDV/range upper bounds from evidence of overlapping values. Use
   measured distinct counts, constraints or an explicit uncertain-overlap fallback.
   
   ## Additional context
   
   Both scan row counts are exact: 150,000 customers and 1,500,000 orders. This
   removes the extra predicates from TPC-H Q22.
   
   Part of #25610.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Avoid inferring full anti-join coverage from range bounds [datafusion]

["LDM-A (via GitHub)" <gi...@apache.org>]

LDM-A commented on issue #25617:
URL: https://github.com/apache/datafusion/issues/25617#issuecomment-6163792708

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org