You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ShashidharM0118 (via GitHub)" <gi...@apache.org> on 2025/12/20 07:32:23 UTC
[PR] feat: output statistics for constant columns in projections [datafusion]
ShashidharM0118 opened a new pull request, #19419:
URL: https://github.com/apache/datafusion/pull/19419
## Which issue does this PR close?
- Closes #19416
## Rationale for this change
Currently, when projecting literal values (constants) in projection operations, the statistics are set to unknown (`ColumnStatistics::new_unknown()`). This prevents the optimizer from using constant column information for various optimizations, such as:
- Sort elimination when sorting by constant columns
## What changes are included in this PR?
1. **Updated `project_statistics` method** in `ProjectionExprs`:
- Added detection and statistics calculation for `Literal` expressions (non-null constants)
- Statistics calculated: `min_value = max_value = literal value`, `distinct_count = 1`, `null_count = 0`, `byte_size = data_type_width × num_rows` (for primitive types)
- NULL literals return `ColumnStatistics::new_unknown()`
2. **Added test**:
- `test_project_statistics_with_literal`: Tests non-null literal statistics
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
## Example
**Before this change:**
`SELECT 42 AS status, name FROM users ORDER BY status ` - Statistics for `status` were unknown
- Optimizer couldn't eliminate the sort operation
**After this change:**
`SELECT 42 AS status, name FROM users ORDER BY status` - Statistics show `status` has `min_value = max_value = 42`, `distinct_count = 1`
- Optimizer can eliminate the sort since the column is already sorted (all values are the same)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653188838
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
+ min_value: Precision::Absent,
+ max_value: Precision::Absent,
+ distinct_count: Precision::Exact(1),
+ null_count,
+ sum_value: Precision::Absent,
+ byte_size: Precision::Absent,
+ }
+ } else {
+ let value = literal.value();
+ let distinct_count = Precision::Exact(1);
+ let null_count = Precision::Exact(0);
+
+ let byte_size = if let Some(byte_width) = data_type.primitive_width()
+ {
+ stats.num_rows.multiply(&Precision::Exact(byte_width))
+ } else {
+ // Complex types depend on array encoding, so set to Absent
+ Precision::Absent
Review Comment:
I think this can be Estimated, but not very critical
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "ShashidharM0118 (via GitHub)" <gi...@apache.org>.
ShashidharM0118 commented on PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#issuecomment-3690974265
Hi @alamb, could you please review this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2639011084
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,78 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ // For NULL literals (constant NULL columns), output proper statistics
+ // This enables optimizations like constant column detection and sort elimination
+ // For constant NULL columns:
+ // - null_count = num_rows (all rows are NULL)
+ // - distinct_count = 1 (all NULLs are considered the same)
+ // - min_value/max_value = Absent (NULLs don't have min/max)
+ // - byte_size = Absent (NULLs don't take space in most representations)
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent, // Can't determine null_count without exact row count
+ };
+
+ ColumnStatistics {
+ min_value: Precision::Absent, // NULLs don't have min/max
+ max_value: Precision::Absent,
+ distinct_count: Precision::Exact(1), // All NULLs are considered the same
+ null_count,
+ sum_value: Precision::Absent, // Sum doesn't make sense for NULLs
+ byte_size: Precision::Absent, // NULLs don't take space
+ }
+ } else {
+ // For constant columns (non-null literals), output proper statistics
+ let value = literal.value();
+
+ // For constant columns:
+ // - min_value = max_value = the literal value
+ // - distinct_count = 1
+ // - null_count = 0
+ // - byte_size = calculated from data type and num_rows
+ let distinct_count = Precision::Exact(1);
+ let null_count = Precision::Exact(0);
+
+ // Calculate byte_size: for primitive types, use width * num_rows
+ let byte_size = if let Some(byte_width) = data_type.primitive_width()
+ {
+ stats.num_rows.multiply(&Precision::Exact(byte_width))
+ } else {
+ // For complex types (Utf8, List, etc.), the byte_size when materialized
+ // as an array depends on the array encoding and representation (e.g.,
+ // dictionary encoding, string view arrays), so we conservatively set it to Absent
+ Precision::Absent
+ };
+
+ // Calculate sum_value: for numeric types, sum = value * num_rows
+ // This is useful for optimizations (e.g., cross joins multiply sum_value by row count)
+ let sum_value = if !value.is_null() {
Review Comment:
The null check on line 639 is redundant since this code is already in the `else` branch (line 614) that handles non-null literals. The outer condition at line 593 has already verified that `literal.value().is_null()` is false. This redundant check adds unnecessary complexity and could be removed to simplify the code logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653185753
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
Review Comment:
Since ScalarValues accept None as well, I wonder if these stats can't just all output the same ScalarValue cloned? (I.e., min/max value, sum value)
byte_size can probably be 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "ShashidharM0118 (via GitHub)" <gi...@apache.org>.
ShashidharM0118 commented on PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#issuecomment-3677532619
NULL literals currently return `ColumnStatistics::new_unknown()` - *Question: Would generating statistics for NULL literals provide additional optimization benefits?*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653185753
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
Review Comment:
Since ScalarValues accept None as well, I wonder if these stats can't just all output the same ScalarValue cloned? (I.e., min/max value, sum value)
byte_size can probably be 0
ScalarValue::new_null(...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "ShashidharM0118 (via GitHub)" <gi...@apache.org>.
ShashidharM0118 commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653546497
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
+ min_value: Precision::Absent,
+ max_value: Precision::Absent,
+ distinct_count: Precision::Exact(1),
+ null_count,
+ sum_value: Precision::Absent,
+ byte_size: Precision::Absent,
+ }
+ } else {
+ let value = literal.value();
+ let distinct_count = Precision::Exact(1);
+ let null_count = Precision::Exact(0);
+
+ let byte_size = if let Some(byte_width) = data_type.primitive_width()
+ {
+ stats.num_rows.multiply(&Precision::Exact(byte_width))
+ } else {
+ // Complex types depend on array encoding, so set to Absent
+ Precision::Absent
Review Comment:
I think deriving the estimated byte size for all possible complex types (nested Lists, Structs, Maps, etc.) here would be quite complex and error-prone ?, as we'd need to manually account for offsets and validity bitmaps for each variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2636892055
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -2593,4 +2625,68 @@ pub(crate) mod tests {
Ok(())
}
+
+ #[test]
+ fn test_project_statistics_with_literal() -> Result<()> {
+ let input_stats = get_stats();
+ let input_schema = get_schema();
+
+ // Projection with literal: SELECT 42 AS constant, col0 AS num
+ let projection = ProjectionExprs::new(vec![
+ ProjectionExpr {
+ expr: Arc::new(Literal::new(ScalarValue::Int64(Some(42)))),
+ alias: "constant".to_string(),
+ },
+ ProjectionExpr {
+ expr: Arc::new(Column::new("col0", 0)),
+ alias: "num".to_string(),
+ },
+ ]);
+
+ let output_stats = projection.project_statistics(
+ input_stats,
+ &projection.project_schema(&input_schema)?,
+ )?;
+
+ // Row count should be preserved
+ assert_eq!(output_stats.num_rows, Precision::Exact(5));
+
+ // Should have 2 column statistics
+ assert_eq!(output_stats.column_statistics.len(), 2);
+
+ // First column (literal 42) should have proper constant statistics
+ assert_eq!(
+ output_stats.column_statistics[0].min_value,
+ Precision::Exact(ScalarValue::Int64(Some(42)))
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].max_value,
+ Precision::Exact(ScalarValue::Int64(Some(42)))
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].distinct_count,
+ Precision::Exact(1)
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].null_count,
+ Precision::Exact(0)
+ );
+ // Int64 is 8 bytes, 5 rows = 40 bytes
+ assert_eq!(
+ output_stats.column_statistics[0].byte_size,
+ Precision::Exact(40)
+ );
Review Comment:
The test should validate the sum_value field for the literal column. While setting sum_value to Precision::Absent is a reasonable choice (as mentioned in the comment on line 619), the test should explicitly assert this to ensure the behavior is documented and verified.
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,38 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>()
+ && !literal.value().is_null()
+ {
+ // For constant columns (non-null literals), output proper statistics
+ let value = literal.value();
+ let data_type = expr.data_type(output_schema)?;
+
+ // For constant columns:
+ // - min_value = max_value = the literal value
+ // - distinct_count = 1
+ // - null_count = 0
+ // - byte_size = calculated from data type and num_rows
+ let distinct_count = Precision::Exact(1);
+ let null_count = Precision::Exact(0);
+
+ // Calculate byte_size: for primitive types, use width * num_rows
+ let byte_size = if let Some(byte_width) = data_type.primitive_width() {
+ stats.num_rows.multiply(&Precision::Exact(byte_width))
+ } else {
+ // For complex types (Utf8, List, etc.), we can't calculate exact size
+ // without knowing the actual data, so leave it as Absent
Review Comment:
The comment says "we can't calculate exact size without knowing the actual data", but we do know the actual data (it's in the value variable). However, the issue is that for complex types like Utf8, the byte_size when materialized as an array depends on the array encoding (e.g., dictionary encoding, string view arrays), which makes exact calculation complex. Consider clarifying the comment to say "For complex types, byte_size depends on array encoding and representation, so we conservatively set it to Absent" to better explain the rationale.
```suggestion
// For complex types (Utf8, List, etc.), the byte_size when materialized
// as an array depends on the array encoding and representation, so we
// conservatively set it to Absent
```
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -2593,4 +2625,68 @@ pub(crate) mod tests {
Ok(())
}
+
+ #[test]
+ fn test_project_statistics_with_literal() -> Result<()> {
+ let input_stats = get_stats();
+ let input_schema = get_schema();
+
+ // Projection with literal: SELECT 42 AS constant, col0 AS num
+ let projection = ProjectionExprs::new(vec![
+ ProjectionExpr {
+ expr: Arc::new(Literal::new(ScalarValue::Int64(Some(42)))),
+ alias: "constant".to_string(),
+ },
+ ProjectionExpr {
+ expr: Arc::new(Column::new("col0", 0)),
+ alias: "num".to_string(),
+ },
+ ]);
+
+ let output_stats = projection.project_statistics(
+ input_stats,
+ &projection.project_schema(&input_schema)?,
+ )?;
+
+ // Row count should be preserved
+ assert_eq!(output_stats.num_rows, Precision::Exact(5));
+
+ // Should have 2 column statistics
+ assert_eq!(output_stats.column_statistics.len(), 2);
+
+ // First column (literal 42) should have proper constant statistics
+ assert_eq!(
+ output_stats.column_statistics[0].min_value,
+ Precision::Exact(ScalarValue::Int64(Some(42)))
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].max_value,
+ Precision::Exact(ScalarValue::Int64(Some(42)))
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].distinct_count,
+ Precision::Exact(1)
+ );
+ assert_eq!(
+ output_stats.column_statistics[0].null_count,
+ Precision::Exact(0)
+ );
+ // Int64 is 8 bytes, 5 rows = 40 bytes
+ assert_eq!(
+ output_stats.column_statistics[0].byte_size,
+ Precision::Exact(40)
+ );
+
+ // Second column (col0) should preserve statistics
+ assert_eq!(
+ output_stats.column_statistics[1].distinct_count,
+ Precision::Exact(5)
+ );
+ assert_eq!(
+ output_stats.column_statistics[1].max_value,
+ Precision::Exact(ScalarValue::Int64(Some(21)))
+ );
+
+ Ok(())
+ }
Review Comment:
The test only covers non-null literals with primitive types (Int64). Consider adding test cases for:
1. Null literals - these should return ColumnStatistics::new_unknown() as per line 591 logic
2. Complex type literals (e.g., Utf8) - these should have byte_size set to Precision::Absent as per line 611
These edge cases are important to verify the complete behavior of the literal statistics calculation.
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,38 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>()
+ && !literal.value().is_null()
+ {
+ // For constant columns (non-null literals), output proper statistics
+ let value = literal.value();
+ let data_type = expr.data_type(output_schema)?;
+
+ // For constant columns:
+ // - min_value = max_value = the literal value
+ // - distinct_count = 1
+ // - null_count = 0
+ // - byte_size = calculated from data type and num_rows
+ let distinct_count = Precision::Exact(1);
+ let null_count = Precision::Exact(0);
+
+ // Calculate byte_size: for primitive types, use width * num_rows
+ let byte_size = if let Some(byte_width) = data_type.primitive_width() {
+ stats.num_rows.multiply(&Precision::Exact(byte_width))
+ } else {
+ // For complex types (Utf8, List, etc.), we can't calculate exact size
+ // without knowing the actual data, so leave it as Absent
+ Precision::Absent
+ };
+
+ ColumnStatistics {
+ min_value: Precision::Exact(value.clone()),
+ max_value: Precision::Exact(value.clone()),
+ distinct_count,
+ null_count,
+ sum_value: Precision::Absent, // Sum doesn't make sense for constants
Review Comment:
The comment states "Sum doesn't make sense for constants", but actually the sum of a constant column would be constant_value * num_rows, which is a meaningful and potentially useful statistic (similar to how cross joins multiply sum_value by row count). Consider calculating sum_value as stats.num_rows.multiply(&Precision::Exact(value.clone())) for numeric types, or document why it's intentionally left as Absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "ShashidharM0118 (via GitHub)" <gi...@apache.org>.
ShashidharM0118 commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653221843
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
Review Comment:
Agreed,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#issuecomment-3699284344
@EmilyMatt given you filed https://github.com/apache/datafusion/issues/19416 could you also help review this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "ShashidharM0118 (via GitHub)" <gi...@apache.org>.
ShashidharM0118 commented on code in PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#discussion_r2653221843
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -587,6 +587,54 @@ impl ProjectionExprs {
let expr = &proj_expr.expr;
let col_stats = if let Some(col) = expr.as_any().downcast_ref::<Column>() {
std::mem::take(&mut stats.column_statistics[col.index()])
+ } else if let Some(literal) = expr.as_any().downcast_ref::<Literal>() {
+ // Handle literal expressions (constants) by calculating proper statistics
+ let data_type = expr.data_type(output_schema)?;
+
+ if literal.value().is_null() {
+ let null_count = match stats.num_rows {
+ Precision::Exact(num_rows) => Precision::Exact(num_rows),
+ _ => Precision::Absent,
+ };
+
+ ColumnStatistics {
Review Comment:
Agreed, I've Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#issuecomment-3699620627
@alamb Added a couple of thoughts but nothing that pops up more than that, looks good to me :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19419:
URL: https://github.com/apache/datafusion/pull/19419#issuecomment-3703731165
Thank you @ShashidharM0118 and @EmilyMatt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: output statistics for constant columns in projections [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #19419:
URL: https://github.com/apache/datafusion/pull/19419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org