You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "appletreeisyellow (via GitHub)" <gi...@apache.org> on 2024/02/13 22:14:28 UTC
[PR] Support `A column is known to be entirely NULL` in `PruningPredicate` [arrow-datafusion]
appletreeisyellow opened a new pull request, #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223
β¦row_count THEN false ELSE ... END`## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1529060372
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Review Comment:
Thank you for updating it. Looks better now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528987024
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
+/// be compared using a `CASE` statement to wrap the rewritten predicate to handle
+/// the case where the column `x` is known to be all `NULL`s. Note this
+/// is different from knowing nothing about the column `x`, which confusingly is
+/// encoded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
+///
+/// Original Predicate | Rewritten Predicate
+/// ------------------ | --------------------
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
+///
Review Comment:
> Do you mean if `x_row_count` is available, the rewritten predicate will be different?
The rewritten predicate will be the same no matter `x_row_count` is available or not.
I think the current comment with a separate `x_row_count` example can be misleading π
I'll update the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525201435
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
I don't understand this comment as I don't see any overwriting happening -- instead it is adding a new entry to the required columns and returning the relevant expression, right?
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
+/// be wrapped around the above rewritten predicate to form the final rewritten predicate.
+/// This step is necessary to handle the case where the column `x` is kown to be all nulls,
+/// This is different from knowing nothing about the column `x`, which confusionly is
+/// enconded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
Review Comment:
There are some misspellings in this paragraph and I think we can make it more concise. Here is a suggestion:
```suggestion
/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
/// be compared using a `CASE` statement to wrap the rewritten predicate to handle
/// the case where the column `x` is known to be all `NULL`s. Note this
/// is different from knowing nothing about the column `x`, which confusingly is
/// encoded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
```
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2819,14 +3171,77 @@ mod tests {
let expected_ret = &[false, true, true, true, false];
prune_with_expr(
- // i IS NULL, with actual null statistcs
+ // i IS NULL, with actual null statistics
col("i").is_null(),
&schema,
&statistics,
expected_ret,
);
}
+ #[test]
+ fn prune_int32_column_is_known_all_null() {
+ let (schema, statistics) = int32_setup();
+
+ // Expression "i < 0"
+ // i [-5, 5] ==> some rows could pass (must keep)
+ // i [1, 11] ==> no rows can pass (not keep)
+ // i [-11, -1] ==> all rows must pass (must keep)
+ // i [NULL, NULL] ==> unknown (must keep)
+ // i [1, NULL] ==> no rows can pass (not keep)
+ let expected_ret = &[true, false, true, true, false];
+
+ prune_with_expr(
+ // i < 0
+ col("i").lt(lit(0)),
+ &schema,
+ &statistics,
+ expected_ret,
+ );
+
+ // provide row counts for each column
+ let statistics = statistics.with_row_counts(
+ "i",
+ vec![
+ Some(10), // 10 rows of data
+ Some(9), // 9 rows of data
+ None, // unknown row counts
+ Some(4),
+ Some(10),
+ ],
+ );
+
+ // provide null counts for each column
+ let statistics = statistics.with_null_counts(
+ "i",
+ vec![
+ Some(0), // no nulls
+ Some(1), // 1 null
+ None, // unknown nulls
+ Some(4), // 4 nulls, which is the same as the row counts, i.e. this column is all null (don't keep)
Review Comment:
π
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2819,14 +3171,77 @@ mod tests {
let expected_ret = &[false, true, true, true, false];
prune_with_expr(
- // i IS NULL, with actual null statistcs
+ // i IS NULL, with actual null statistics
col("i").is_null(),
&schema,
&statistics,
expected_ret,
);
}
+ #[test]
+ fn prune_int32_column_is_known_all_null() {
+ let (schema, statistics) = int32_setup();
+
+ // Expression "i < 0"
+ // i [-5, 5] ==> some rows could pass (must keep)
+ // i [1, 11] ==> no rows can pass (not keep)
+ // i [-11, -1] ==> all rows must pass (must keep)
+ // i [NULL, NULL] ==> unknown (must keep)
+ // i [1, NULL] ==> no rows can pass (not keep)
+ let expected_ret = &[true, false, true, true, false];
+
+ prune_with_expr(
+ // i < 0
+ col("i").lt(lit(0)),
+ &schema,
+ &statistics,
+ expected_ret,
+ );
+
+ // provide row counts for each column
+ let statistics = statistics.with_row_counts(
+ "i",
+ vec![
+ Some(10), // 10 rows of data
+ Some(9), // 9 rows of data
+ None, // unknown row counts
+ Some(4),
+ Some(10),
+ ],
+ );
+
Review Comment:
Can you also please test that just providing `row_counts` (but not null counts) change the result?
Something like
```suggestion
// pruning result is still the same if we only know row counts
prune_with_expr(
// i < 0
col("i").lt(lit(0)),
&schema,
&statistics,
expected_ret,
);
```
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2819,14 +3171,77 @@ mod tests {
let expected_ret = &[false, true, true, true, false];
prune_with_expr(
- // i IS NULL, with actual null statistcs
+ // i IS NULL, with actual null statistics
col("i").is_null(),
&schema,
&statistics,
expected_ret,
);
}
+ #[test]
+ fn prune_int32_column_is_known_all_null() {
+ let (schema, statistics) = int32_setup();
+
+ // Expression "i < 0"
+ // i [-5, 5] ==> some rows could pass (must keep)
+ // i [1, 11] ==> no rows can pass (not keep)
+ // i [-11, -1] ==> all rows must pass (must keep)
+ // i [NULL, NULL] ==> unknown (must keep)
+ // i [1, NULL] ==> no rows can pass (not keep)
+ let expected_ret = &[true, false, true, true, false];
+
+ prune_with_expr(
+ // i < 0
+ col("i").lt(lit(0)),
+ &schema,
+ &statistics,
+ expected_ret,
+ );
+
+ // provide row counts for each column
+ let statistics = statistics.with_row_counts(
+ "i",
+ vec![
+ Some(10), // 10 rows of data
+ Some(9), // 9 rows of data
+ None, // unknown row counts
+ Some(4),
+ Some(10),
+ ],
+ );
+
+ // provide null counts for each column
+ let statistics = statistics.with_null_counts(
+ "i",
+ vec![
+ Some(0), // no nulls
+ Some(1), // 1 null
+ None, // unknown nulls
+ Some(4), // 4 nulls, which is the same as the row counts, i.e. this column is all null (don't keep)
+ Some(0), // 0 nulls (max=null too which means no known max)
+ ],
+ );
+
+ // Expression "i < 0" with actual null and row counts statistics
+ // col | min, max | row counts | null counts |
+ // ----+--------------+------------+-------------+
+ // i | [-5, 5] | 10 | 0 | ==> Some rows could pass (must keep)
+ // i | [1, 11] | 9 | 1 | ==> No rows can pass (not keep)
+ // i | [-11,-1] | Unknown | Unknown | ==> All rows must pass (must keep)
+ // i | [NULL, NULL] | 4 | 4 | ==> The column is all null (not keep)
+ // i | [1, NULL] | 10 | 0 | ==> No rows can pass (not keep)
+ let expected_ret = &[true, false, true, false, false];
Review Comment:
the 4th element in this array is `false` which is different than the 4th element when null counts aren't known . π
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -326,28 +348,46 @@ pub trait PruningStatistics {
/// LiteralGuarantees are not satisfied
///
/// **Second Pass**: Evaluates the rewritten expression using the
-/// min/max/null_counts values for each column for each container. For any
+/// min/max/null_counts/row_counts values for each column for each container. For any
/// container that this expression evaluates to `false`, it rules out those
/// containers.
///
-/// For example, given the predicate, `x = 5 AND y = 10`, if we know `x` is
-/// between `1 and 100` and we know that `y` is between `4` and `7`, the input
-/// statistics might look like
+///
+/// ### Example 1
+/// Given the predicate, `x = 5 AND y = 10`, if we know that for a given container, `x` is
+/// between `1 and 100` and we know that `y` is between `4` and `7`, we know nothing about
+/// the null count and row count of `x` and `y`, the input statistics might look like:
///
/// Column | Value
/// -------- | -----
/// `x_min` | `1`
/// `x_max` | `100`
+/// `x_null_count` | `null`
+/// `x_row_count` | `null`
/// `y_min` | `4`
/// `y_max` | `7`
+/// `y_null_count` | `null`
+/// `y_row_count` | `null`
///
/// The rewritten predicate would look like
///
-/// `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
+/// ```sql
+/// CASE
+/// WHEN x_null_count = x_row_count THEN false
+/// ELSE x_min <= 5 AND 5 <= x_max
+/// END
+/// AND
+/// CASE
+/// WHEN y_null_count = y_row_count THEN false
+/// ELSE y_min <= 10 AND 10 <= y_max
+/// END
+/// ```
///
-/// When these values are substituted in to the rewritten predicate and
+/// When these statistics values are substituted in to the rewritten predicate and
/// simplified, the result is `false`:
///
+/// * `CASE WHEN null = null THEN false ELSE 1 <= 5 AND 5 <= 100 END AND CASE WHEN null = null THEN false ELSE 4 <= 10 AND 10 <= 7 END`
+/// * `null = null` is `null` which is false, so the `CASE` expression will use the `ELSE` clause
Review Comment:
```suggestion
/// * `null = null` is `null` which is is not true, so the `CASE` expression will use the `ELSE` clause
```
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
+ fn null_count_column_expr(&mut self) -> Result<Arc<dyn PhysicalExpr>> {
+ // overwrite to [`phys_expr::Column`]
+ let column_expr = Arc::new(self.column.clone()) as _;
+
+ // null_count is DataType::UInt64, which is different from the column's data type (i.e. self.field)
+ let null_count_field = &Field::new(self.field.name(), DataType::UInt64, true);
+
+ self.required_columns.null_count_column_expr(
+ &self.column,
+ &column_expr,
+ null_count_field,
+ )
+ }
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
Review Comment:
I have the same question about this comment
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Ok(statistics_expr)
}
+/// Wrap the statistics expression in a case expression.
+/// This is necessary to handle the case where the column is known
+/// to be all nulls.
+///
+/// For example:
+///
+/// `x_min <= 10 AND 10 <= x_max`
+///
+/// will become
+///
+/// ```sql
+/// CASE
+/// WHEN x_null_count = x_row_count THEN false
+/// ELSE x_min <= 10 AND 10 <= x_max
+/// END
+/// ````
+///
+/// If the column is known to be all nulls, then the expression
+/// `x_null_count = x_row_count` will be true, which will cause the
+/// case expression to return false. Therefore, prune out the container.
+fn wrap_case_expr(
Review Comment:
I wonder if this method might be easier to find if it were on the `PruningExpressionBuilder` itself?
Like
```rust
impl PruningExpressionBuilder {
...
fn wrap_case_expr(&mut self, statistics_expr: Arc<dyn PhysicalExpr>) -> Result<Arc<dyn PhysicalExpr>> {
...
}
```
##########
datafusion/sqllogictest/test_files/repartition_scan.slt:
##########
@@ -138,7 +138,7 @@ physical_plan
SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
--CoalesceBatchesExec: target_batch_size=8192
----FilterExec: column1@0 != 42
-------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1, required_guarantees=[column1 not in (42)]
+------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)]
Review Comment:
π€ now that the pruning predicate is getting more complex, perhaps we should not display it by default anymore in explain plans. Maybe we can add a config option (as a follow on PR) that is disabled by default π€ to control if it is displayed
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -364,6 +404,50 @@ pub trait PruningStatistics {
/// more analysis, for example by actually reading the data and evaluating the
/// predicate row by row.
///
+/// ### Example 2
+/// Given the same predicate, `x = 5 AND y = 10`, if we know that for another given container,
+/// `x_min` is NULL and `x_max` is NULL, `x_null_count` is `100` and `x_row_count` is `100`;
Review Comment:
```suggestion
/// `x_min` is NULL and `x_max` is NULL (the min/max values are unknown), `x_null_count` is `100` and `x_row_count` is `100`;
```
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2066,7 +2301,21 @@ mod tests {
let expr = col("c1")
.lt(lit(1))
.and(col("c2").eq(lit(2)).or(col("c2").eq(lit(3))));
- let expected_expr = "c1_min@0 < 1 AND (c2_min@1 <= 2 AND 2 <= c2_max@2 OR c2_min@1 <= 3 AND 3 <= c2_max@2)";
+ let expected_expr = "\
+ CASE \
+ WHEN c1_null_count@1 = c1_row_count@2 THEN false \
+ ELSE c1_min@0 < 1 \
+ END \
+ AND (\
+ CASE \
+ WHEN c2_null_count@5 = c2_row_count@6 THEN false \
Review Comment:
It would be nice in the future to combine these clauses (so we didn't have the repeated CASE expression) but for now I think this is good enought
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1554847981
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -306,17 +315,23 @@ pub trait PruningStatistics {
/// * `false`: there are no rows that could possibly match the predicate,
/// **PRUNES** the container
///
-/// For example, given a column `x`, the `x_min` and `x_max` and `x_null_count`
-/// represent the minimum and maximum values, and the null count of column `x`,
-/// provided by the `PruningStatistics`. Here are some examples of the rewritten
-/// predicates:
+/// For example, given a column `x`, the `x_min`, `x_max`, `x_null_count`, and
+/// `x_row_count` represent the minimum and maximum values, the null count of
+/// column `x`, and the row count of column `x`, provided by the `PruningStatistics`.
+/// `x_null_count` and `x_row_count` are used to handle the case where the column `x`
+/// is known to be all `NULL`s. Note this is different from knowing nothing about
+/// the column `x`, which confusingly is encoded by returning `NULL` for the min/max
+/// values from [`PruningStatistics::max_values`] and [`PruningStatistics::min_values`].
+///
+/// Here are some examples of the rewritten predicates:
///
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
-/// `x = 5` | `x_min <= 5 AND 5 <= x_max`
-/// `x < 5` | `x_max < 5`
-/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
-/// `x IS NULL` | `x_null_count > 0`
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
Review Comment:
https://github.com/apache/arrow-datafusion/blob/fa7ca27c15328247dbf98b2f8773c19398b8a745/datafusion/core/src/physical_optimizer/pruning.rs#L1285-L1288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1529038563
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
+/// be compared using a `CASE` statement to wrap the rewritten predicate to handle
+/// the case where the column `x` is known to be all `NULL`s. Note this
+/// is different from knowing nothing about the column `x`, which confusingly is
+/// encoded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
+///
+/// Original Predicate | Rewritten Predicate
+/// ------------------ | --------------------
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
+///
Review Comment:
Updated in Updated in https://github.com/apache/arrow-datafusion/pull/9223/commits/a8639fb8aceb12333afe99c9fd6275077a017c6e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1555022210
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -306,17 +315,23 @@ pub trait PruningStatistics {
/// * `false`: there are no rows that could possibly match the predicate,
/// **PRUNES** the container
///
-/// For example, given a column `x`, the `x_min` and `x_max` and `x_null_count`
-/// represent the minimum and maximum values, and the null count of column `x`,
-/// provided by the `PruningStatistics`. Here are some examples of the rewritten
-/// predicates:
+/// For example, given a column `x`, the `x_min`, `x_max`, `x_null_count`, and
+/// `x_row_count` represent the minimum and maximum values, the null count of
+/// column `x`, and the row count of column `x`, provided by the `PruningStatistics`.
+/// `x_null_count` and `x_row_count` are used to handle the case where the column `x`
+/// is known to be all `NULL`s. Note this is different from knowing nothing about
+/// the column `x`, which confusingly is encoded by returning `NULL` for the min/max
+/// values from [`PruningStatistics::max_values`] and [`PruningStatistics::min_values`].
+///
+/// Here are some examples of the rewritten predicates:
///
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
-/// `x = 5` | `x_min <= 5 AND 5 <= x_max`
-/// `x < 5` | `x_max < 5`
-/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
-/// `x IS NULL` | `x_null_count > 0`
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
Review Comment:
update: https://github.com/apache/arrow-datafusion/pull/9986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525511917
##########
datafusion/sqllogictest/test_files/repartition_scan.slt:
##########
@@ -138,7 +138,7 @@ physical_plan
SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
--CoalesceBatchesExec: target_batch_size=8192
----FilterExec: column1@0 != 42
-------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1, required_guarantees=[column1 not in (42)]
+------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)]
Review Comment:
π Tracked in https://github.com/apache/arrow-datafusion/issues/9171#issuecomment-1998518531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525632419
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
If the query predicate is `WHERE cast(x AS data_type)`, the original rewrite will turn `x_min` into `cast(x_min)` and `x_max` into `cast(x_max AS data_type)`. Same for `try_cast()`. See here:
https://github.com/appletreeisyellow/arrow-datafusion/blob/chunchun/pruning-predicate-column-known-tobe-null/datafusion/core/src/physical_optimizer/pruning.rs#L1075-L1101
We don't want `cast()` and `try_cast()` to rewrite `x_null_count` into `cast(x_null_count AS data_type)`, so this step is to avoid `cast(x_null_count AS data_type)` from happening
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
+ fn null_count_column_expr(&mut self) -> Result<Arc<dyn PhysicalExpr>> {
+ // overwrite to [`phys_expr::Column`]
+ let column_expr = Arc::new(self.column.clone()) as _;
+
+ // null_count is DataType::UInt64, which is different from the column's data type (i.e. self.field)
+ let null_count_field = &Field::new(self.field.name(), DataType::UInt64, true);
+
+ self.required_columns.null_count_column_expr(
+ &self.column,
+ &column_expr,
+ null_count_field,
+ )
+ }
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
Review Comment:
See https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525632419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525511859
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2066,7 +2301,21 @@ mod tests {
let expr = col("c1")
.lt(lit(1))
.and(col("c2").eq(lit(2)).or(col("c2").eq(lit(3))));
- let expected_expr = "c1_min@0 < 1 AND (c2_min@1 <= 2 AND 2 <= c2_max@2 OR c2_min@1 <= 3 AND 3 <= c2_max@2)";
+ let expected_expr = "\
+ CASE \
+ WHEN c1_null_count@1 = c1_row_count@2 THEN false \
+ ELSE c1_min@0 < 1 \
+ END \
+ AND (\
+ CASE \
+ WHEN c2_null_count@5 = c2_row_count@6 THEN false \
Review Comment:
π Tracked in https://github.com/apache/arrow-datafusion/issues/9171#issuecomment-1998518531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527196814
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
Review Comment:
It is better to mention `x_row_count` in the beginning above, i.e.,
```
/// For example, given a column `x`, the `x_min` and `x_max`, `x_null_count` and `x_row_count`
/// represent the minimum and maximum values, and the null count of column `x`,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527196814
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
Review Comment:
It is better to mention `x_row_count` in the beginning above, i.e.,
```
/// For example, given a column `x`, the `x_min` and `x_max`, `x_null_count` and `x_row_count`
/// represent the minimum and maximum values, the null count of column `x`, and the row count of column `x`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#issuecomment-2005206304
Thank you @appletreeisyellow and @viirya for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527196025
##########
datafusion/sqllogictest/test_files/repartition_scan.slt:
##########
@@ -138,7 +138,7 @@ physical_plan
SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
--CoalesceBatchesExec: target_batch_size=8192
----FilterExec: column1@0 != 42
-------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1, required_guarantees=[column1 not in (42)]
+------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)]
Review Comment:
By default, we can show part of pruning predicate (i.e., truncated one) so users can know there is pruning predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527203857
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Review Comment:
If there is no null_count or row_count statistics, do we still need to rewrite it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1554837787
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -306,17 +315,23 @@ pub trait PruningStatistics {
/// * `false`: there are no rows that could possibly match the predicate,
/// **PRUNES** the container
///
-/// For example, given a column `x`, the `x_min` and `x_max` and `x_null_count`
-/// represent the minimum and maximum values, and the null count of column `x`,
-/// provided by the `PruningStatistics`. Here are some examples of the rewritten
-/// predicates:
+/// For example, given a column `x`, the `x_min`, `x_max`, `x_null_count`, and
+/// `x_row_count` represent the minimum and maximum values, the null count of
+/// column `x`, and the row count of column `x`, provided by the `PruningStatistics`.
+/// `x_null_count` and `x_row_count` are used to handle the case where the column `x`
+/// is known to be all `NULL`s. Note this is different from knowing nothing about
+/// the column `x`, which confusingly is encoded by returning `NULL` for the min/max
+/// values from [`PruningStatistics::max_values`] and [`PruningStatistics::min_values`].
+///
+/// Here are some examples of the rewritten predicates:
///
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
-/// `x = 5` | `x_min <= 5 AND 5 <= x_max`
-/// `x < 5` | `x_max < 5`
-/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
-/// `x IS NULL` | `x_null_count > 0`
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
Review Comment:
@appletreeisyellow @alamb Sorry i am confused here, i think here just need `x IS NULL` | ` x_null_count > 0 END`
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -306,17 +315,23 @@ pub trait PruningStatistics {
/// * `false`: there are no rows that could possibly match the predicate,
/// **PRUNES** the container
///
-/// For example, given a column `x`, the `x_min` and `x_max` and `x_null_count`
-/// represent the minimum and maximum values, and the null count of column `x`,
-/// provided by the `PruningStatistics`. Here are some examples of the rewritten
-/// predicates:
+/// For example, given a column `x`, the `x_min`, `x_max`, `x_null_count`, and
+/// `x_row_count` represent the minimum and maximum values, the null count of
+/// column `x`, and the row count of column `x`, provided by the `PruningStatistics`.
+/// `x_null_count` and `x_row_count` are used to handle the case where the column `x`
+/// is known to be all `NULL`s. Note this is different from knowing nothing about
+/// the column `x`, which confusingly is encoded by returning `NULL` for the min/max
+/// values from [`PruningStatistics::max_values`] and [`PruningStatistics::min_values`].
+///
+/// Here are some examples of the rewritten predicates:
///
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
-/// `x = 5` | `x_min <= 5 AND 5 <= x_max`
-/// `x < 5` | `x_max < 5`
-/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
-/// `x IS NULL` | `x_null_count > 0`
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
Review Comment:
@appletreeisyellow @alamb Sorry i am confused here, i think here just need `x IS NULL` | ` x_null_count > 0 END` π€
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525628161
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -2819,14 +3171,77 @@ mod tests {
let expected_ret = &[false, true, true, true, false];
prune_with_expr(
- // i IS NULL, with actual null statistcs
+ // i IS NULL, with actual null statistics
col("i").is_null(),
&schema,
&statistics,
expected_ret,
);
}
+ #[test]
+ fn prune_int32_column_is_known_all_null() {
+ let (schema, statistics) = int32_setup();
+
+ // Expression "i < 0"
+ // i [-5, 5] ==> some rows could pass (must keep)
+ // i [1, 11] ==> no rows can pass (not keep)
+ // i [-11, -1] ==> all rows must pass (must keep)
+ // i [NULL, NULL] ==> unknown (must keep)
+ // i [1, NULL] ==> no rows can pass (not keep)
+ let expected_ret = &[true, false, true, true, false];
+
+ prune_with_expr(
+ // i < 0
+ col("i").lt(lit(0)),
+ &schema,
+ &statistics,
+ expected_ret,
+ );
+
+ // provide row counts for each column
+ let statistics = statistics.with_row_counts(
+ "i",
+ vec![
+ Some(10), // 10 rows of data
+ Some(9), // 9 rows of data
+ None, // unknown row counts
+ Some(4),
+ Some(10),
+ ],
+ );
+
Review Comment:
Added in https://github.com/apache/arrow-datafusion/pull/9223/commits/a385bffe572653a37868fcca7f464e1b3b6cb5ff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#issuecomment-1964468245
The major feature is implemented, but there are two things I plan to do before I open it for review:
1. Add more doc
2. Test against IOx
I plan to continue the work on the week of March 4, 2024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#issuecomment-2001966346
I plan to merge this PR on Monday unless anyone else would like more time to comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527197640
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
+/// be compared using a `CASE` statement to wrap the rewritten predicate to handle
+/// the case where the column `x` is known to be all `NULL`s. Note this
+/// is different from knowing nothing about the column `x`, which confusingly is
+/// encoded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
+///
+/// Original Predicate | Rewritten Predicate
+/// ------------------ | --------------------
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
+///
Review Comment:
Hmm, I'm confused by this section of rewritten predicate. These predicates are the same as above, but their rewritten predicates are different. Do you mean if `x_row_count` is available, the rewritten predicate will be different?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528939357
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Review Comment:
Hmm, then I think the rules order could be changed? It sounds not making sense to rewrite predicates based on statistics before the statistics are ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528950313
##########
datafusion/sqllogictest/test_files/repartition_scan.slt:
##########
@@ -138,7 +138,7 @@ physical_plan
SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
--CoalesceBatchesExec: target_batch_size=8192
----FilterExec: column1@0 != 42
-------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1, required_guarantees=[column1 not in (42)]
+------ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..207], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:207..414], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:202..405]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)]
Review Comment:
I like the idea of showing truncated pruning predicate by default. I updated this idea in https://github.com/apache/arrow-datafusion/issues/9171#issuecomment-1998518531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1525651339
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Ok(statistics_expr)
}
+/// Wrap the statistics expression in a case expression.
+/// This is necessary to handle the case where the column is known
+/// to be all nulls.
+///
+/// For example:
+///
+/// `x_min <= 10 AND 10 <= x_max`
+///
+/// will become
+///
+/// ```sql
+/// CASE
+/// WHEN x_null_count = x_row_count THEN false
+/// ELSE x_min <= 10 AND 10 <= x_max
+/// END
+/// ````
+///
+/// If the column is known to be all nulls, then the expression
+/// `x_null_count = x_row_count` will be true, which will cause the
+/// case expression to return false. Therefore, prune out the container.
+fn wrap_case_expr(
Review Comment:
I will consider it in a follow-up PR π
Tracked in https://github.com/apache/arrow-datafusion/issues/9171#issuecomment-1998518531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1529040789
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Review Comment:
My bad, I think the doc I wrote was not clear, I updated the doc in https://github.com/apache/arrow-datafusion/pull/9223/commits/a8639fb8aceb12333afe99c9fd6275077a017c6e. What do you think now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527203613
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
I feel that the current comment is confused. It is better to rephrase it a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528905344
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -1320,14 +1457,56 @@ fn build_statistics_expr(
);
}
};
+ let statistics_expr = wrap_case_expr(statistics_expr, expr_builder)?;
Review Comment:
Theoretically, if there is no `null_count` or `row_count` statistics, we don't need to rewrite it. Practically, we don't know about `null_count` or `row_count` statistics when the rewrite takes place, because the rewrite happens in `PruningPredicate` first and then `null_count` and `row_count` statistics come in later in `PruningStatistics`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528987024
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
+/// be compared using a `CASE` statement to wrap the rewritten predicate to handle
+/// the case where the column `x` is known to be all `NULL`s. Note this
+/// is different from knowing nothing about the column `x`, which confusingly is
+/// encoded by returning `NULL` for the min/max values from [`PruningStatistics::min_values`].
+///
+/// Original Predicate | Rewritten Predicate
+/// ------------------ | --------------------
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
+///
Review Comment:
> Do you mean if `x_row_count` is available, the rewritten predicate will be different?
The rewritten predicate will be the same no matter `x_row_count` is available or not.
I think the current comment with a separate `x_row_count` example can be misleading. I'll update the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527203388
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
I checked how this function is used below. I think it is not used to rewrite predicate expression. It is to actually simply retune the null_count physical expression no matter the predicate expression is, i.e., `x > 5` => `null_count_x`, `cast(x as int) < 10` => `null_count_x`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1528942393
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
Yeah, I was also confused by the comment I wrote a month ago π
I rephrased comments in https://github.com/apache/arrow-datafusion/pull/9223/commits/275ddc6ea1e30c9e453d9f4c3fece57b835dab1f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1529036545
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -326,28 +348,46 @@ pub trait PruningStatistics {
/// LiteralGuarantees are not satisfied
///
/// **Second Pass**: Evaluates the rewritten expression using the
-/// min/max/null_counts values for each column for each container. For any
+/// min/max/null_counts/row_counts values for each column for each container. For any
/// container that this expression evaluates to `false`, it rules out those
/// containers.
///
-/// For example, given the predicate, `x = 5 AND y = 10`, if we know `x` is
-/// between `1 and 100` and we know that `y` is between `4` and `7`, the input
-/// statistics might look like
+///
+/// ### Example 1
+/// Given the predicate, `x = 5 AND y = 10`, if we know that for a given container, `x` is
+/// between `1 and 100` and we know that `y` is between `4` and `7`, we know nothing about
+/// the null count and row count of `x` and `y`, the input statistics might look like:
///
/// Column | Value
/// -------- | -----
/// `x_min` | `1`
/// `x_max` | `100`
+/// `x_null_count` | `null`
+/// `x_row_count` | `null`
/// `y_min` | `4`
/// `y_max` | `7`
+/// `y_null_count` | `null`
+/// `y_row_count` | `null`
///
/// The rewritten predicate would look like
///
-/// `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
+/// ```sql
+/// CASE
+/// WHEN x_null_count = x_row_count THEN false
+/// ELSE x_min <= 5 AND 5 <= x_max
+/// END
+/// AND
+/// CASE
+/// WHEN y_null_count = y_row_count THEN false
+/// ELSE y_min <= 10 AND 10 <= y_max
+/// END
+/// ```
Review Comment:
> I wonder why we need to put x_null_count and x_row_count into rewritten predicate if we know nothing about them?
That's my bad that the current doc is not clear. The rewritten predicate should happen first for physical plan, then in the pruning stage, we get `x_null_count` and `x_row_count` statistics for each container (row group, file, etc). So no matter whether we know about x_null_count and x_row_count or not, the rewritten predicate happens first.
Updated in https://github.com/apache/arrow-datafusion/pull/9223/commits/a8639fb8aceb12333afe99c9fd6275077a017c6e
> Previous rewritten predicate looks more correct/concise to me.
Yes, totally agree. I plan to simplify the case expression and make it easy to read in a follow-up PR, tracked in https://github.com/apache/arrow-datafusion/issues/9171#issuecomment-1998518531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1555021735
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -306,17 +315,23 @@ pub trait PruningStatistics {
/// * `false`: there are no rows that could possibly match the predicate,
/// **PRUNES** the container
///
-/// For example, given a column `x`, the `x_min` and `x_max` and `x_null_count`
-/// represent the minimum and maximum values, and the null count of column `x`,
-/// provided by the `PruningStatistics`. Here are some examples of the rewritten
-/// predicates:
+/// For example, given a column `x`, the `x_min`, `x_max`, `x_null_count`, and
+/// `x_row_count` represent the minimum and maximum values, the null count of
+/// column `x`, and the row count of column `x`, provided by the `PruningStatistics`.
+/// `x_null_count` and `x_row_count` are used to handle the case where the column `x`
+/// is known to be all `NULL`s. Note this is different from knowing nothing about
+/// the column `x`, which confusingly is encoded by returning `NULL` for the min/max
+/// values from [`PruningStatistics::max_values`] and [`PruningStatistics::min_values`].
+///
+/// Here are some examples of the rewritten predicates:
///
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
-/// `x = 5` | `x_min <= 5 AND 5 <= x_max`
-/// `x < 5` | `x_max < 5`
-/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
-/// `x IS NULL` | `x_null_count > 0`
+/// `x = 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END`
+/// `x < 5` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_max < 5 END`
+/// `x = 5 AND y = 10` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_min <= 5 AND 5 <= x_max END AND CASE WHEN y_null_count = y_row_count THEN false ELSE y_min <= 10 AND 10 <= y_max END`
+/// `x IS NULL` | `CASE WHEN x_null_count = x_row_count THEN false ELSE x_null_count > 0 END`
Review Comment:
I think you are right @Ted-Jiang. Nicely spotted -- I double checked and indeed all that is done is `null_count > 0`. I'll a PR to fix.
```
β― explain select duration_nano from traces where duration_nano IS NULL;
| | ParquetExec: file_groups={16 groups: [...]}, projection=[duration_nano], predicate=duration_nano@1 IS NULL, pruning_predicate=duration_nano_null_count@0 > 0, required_guarantees=[] |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#issuecomment-1995855657
Picking up this PR again. I'm able to test this branch against IOx by using the new `PruningStatistics::row_counts()` and verifying that `PruningPredicate` is able to prune out containers that have columns with all NULL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#issuecomment-1997457318
I plan to review this PR later today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527203388
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
I checked how this function is used below. I think it is not used to rewrite predicate expression. It is to actually simply retune the null_count physical expression no matter the predicate expression is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527202192
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -903,6 +1004,42 @@ impl<'a> PruningExpressionBuilder<'a> {
self.required_columns
.max_column_expr(&self.column, &self.column_expr, self.field)
}
+
+ /// Note that this function intentionally overwrites the column expression to [`phys_expr::Column`].
+ /// i.e. expressions like [`phys_expr::CastExpr`] or [`phys_expr::TryCastExpr`] will be overwritten.
+ ///
+ /// This is to avoid cases like `cast(x_null_count)` or `try_cast(x_null_count)`.
Review Comment:
Hmm, I don't quite get this part too. Why we need to handle `cast` or `try_cast` here specially? If the predicate is `cast(x as dt)`, shouldn't we keep it as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1527197990
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -326,28 +348,46 @@ pub trait PruningStatistics {
/// LiteralGuarantees are not satisfied
///
/// **Second Pass**: Evaluates the rewritten expression using the
-/// min/max/null_counts values for each column for each container. For any
+/// min/max/null_counts/row_counts values for each column for each container. For any
/// container that this expression evaluates to `false`, it rules out those
/// containers.
///
-/// For example, given the predicate, `x = 5 AND y = 10`, if we know `x` is
-/// between `1 and 100` and we know that `y` is between `4` and `7`, the input
-/// statistics might look like
+///
+/// ### Example 1
+/// Given the predicate, `x = 5 AND y = 10`, if we know that for a given container, `x` is
+/// between `1 and 100` and we know that `y` is between `4` and `7`, we know nothing about
+/// the null count and row count of `x` and `y`, the input statistics might look like:
///
/// Column | Value
/// -------- | -----
/// `x_min` | `1`
/// `x_max` | `100`
+/// `x_null_count` | `null`
+/// `x_row_count` | `null`
/// `y_min` | `4`
/// `y_max` | `7`
+/// `y_null_count` | `null`
+/// `y_row_count` | `null`
///
/// The rewritten predicate would look like
///
-/// `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
+/// ```sql
+/// CASE
+/// WHEN x_null_count = x_row_count THEN false
+/// ELSE x_min <= 5 AND 5 <= x_max
+/// END
+/// AND
+/// CASE
+/// WHEN y_null_count = y_row_count THEN false
+/// ELSE y_min <= 10 AND 10 <= y_max
+/// END
+/// ```
Review Comment:
I wonder why we need to put x_null_count and x_row_count into rewritten predicate if we know nothing about them? Previous rewritten predicate looks more correct/concise to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support "A column is known to be entirely NULL" in `PruningPredicate` [arrow-datafusion]
Posted by "appletreeisyellow (via GitHub)" <gi...@apache.org>.
appletreeisyellow commented on code in PR #9223:
URL: https://github.com/apache/arrow-datafusion/pull/9223#discussion_r1529038295
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -318,6 +327,19 @@ pub trait PruningStatistics {
/// `x = 5 AND y = 10` | `x_min <= 5 AND 5 <= x_max AND y_min <= 10 AND 10 <= y_max`
/// `x IS NULL` | `x_null_count > 0`
///
+/// In addition, for a given column `x`, the `x_null_count` and `x_row_count` will
Review Comment:
Updated in Updated in https://github.com/apache/arrow-datafusion/pull/9223/commits/a8639fb8aceb12333afe99c9fd6275077a017c6e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org