You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "friendlymatthew (via GitHub)" <gi...@apache.org> on 2026/03/10 18:35:31 UTC
[PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
friendlymatthew opened a new pull request, #20854:
URL: https://github.com/apache/datafusion/pull/20854
## Which issue does this PR close?
- Related https://github.com/apache/datafusion/pull/20822
- Closes https://github.com/apache/datafusion/issues/20603
## Rationale for this change
This PR refines how the `FilterCandidateBuilder` projects struct columns during Parquet row filter pushdown.
Previously, a filter like `s['value'] > 10` would cause the reader to decode all leaf columns of a struct `s`, because `PushdownChecker` only tracked the root column index and expanded it to every leaf. This wastes I/O and decode time on fields the filter never touches
Now, the builder resolves only the matching Parquet leaf columns. It does this by building a pruned filter schema that reflects exactly what the Parquet reader produces when projecting a subset of struct leaves, ensuring the expression evaluates against the correct types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2918863264
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
sort and dedup are necessary since we concatenate `leaf_indices_for_roots` and `resolve_struct_field_leaves`. The first iterates parquet leaves 0..N collecting those belonging to regular (non-struct) cols, and the second does the same for struct field accesses. Both produce individually sorted output, but when a struct column appears before a regular column in the schema, the struct's leaf indices are numerically lower.
Dedup is needed because the same struct field can appear multiple times in a filter expression like `get_field(s, 'val') > 5 and get_field(s, 'val') < 100` producing duplicate entries in `struct_field_accesses`. Without dedup, we'd double count the compressed size of that column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914211709
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
Will this result in subtle bugs that assert on the column order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2913765088
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -449,7 +466,17 @@ impl TreeNodeVisitor<'_> for PushdownChecker<'_> {
{
let return_type = func.return_type();
if !DataType::is_nested(return_type) {
- if let Some(recursion) = self.check_struct_field_column(column.name())
+ let field_path = args[1..]
+ .iter()
+ .filter_map(|arg| {
+ arg.as_any().downcast_ref::<Literal>().and_then(|lit| {
+ lit.value().try_as_str().flatten().map(|s| s.to_string())
+ })
+ })
+ .collect();
Review Comment:
This silently skips non-literal or non-string arguments. In practice, this is safe because `GetFieldFunc` always takes string literal field names, and the simplifier `GetFieldFunc::simplify` flattens chained `get_field` calls into a single call before we reach physical planning
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
};
- self.required_columns.push(idx);
+ self.struct_field_accesses.push(StructFieldAccess {
Review Comment:
struct field accesses are tracked in a separate vec rather than being pushed into `required_columns`
This is intentional since `required_columns` feeds into `leaf_indices_for_roots` which expands a root index to all its leaves. By keeping the struct accesses separate, we can resolve them to only the specific leaves needed via `resolve_struct_field_leaves`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2926455797
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
I don't think that's true. We resolve partition columns to literals before we reach this point. And any missing columns would have also been replaced with null literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914211709
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
Will this result in subtle bugs that assert on the column order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2926464080
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -519,6 +562,149 @@ fn leaf_indices_for_roots(
.collect()
}
+/// Resolves struct field access to specific Parquet leaf column indices
+///
+/// For every `StructFieldAccess`, finds the leaf columns in the Parquet schema
+/// whose path matches the struct root name + field path. This avoids reading all
+/// leaves of a struct when only specific fields are needed
+fn resolve_struct_field_leaves(
+ accesses: &[StructFieldAccess],
+ file_schema: &Schema,
+ schema_descr: &SchemaDescriptor,
+) -> Vec<usize> {
+ let mut leaf_indices = Vec::new();
+
+ for access in accesses {
+ let root_name = file_schema.field(access.root_index).name();
+ let prefix = std::iter::once(root_name.as_str())
+ .chain(access.field_path.iter().map(|p| p.as_str()))
+ .collect::<Vec<_>>();
+
+ for leaf_idx in 0..schema_descr.num_columns() {
+ let col = schema_descr.column(leaf_idx);
+ let col_path = col.path().parts();
+
+ // A leaf matches if its path starts with our prefix.
+ // e.g., prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+
+ // a leaf matches iff its path starts with our prefix
+ // for example: prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+ let leaf_matches_path = col_path.len() >= prefix.len()
+ && col_path.iter().zip(prefix.iter()).all(|(a, b)| a == b);
+
+ if leaf_matches_path {
+ leaf_indices.push(leaf_idx);
+ }
+ }
+ }
+
+ leaf_indices
+}
+
+/// Builds a filter schema that includes only the fields actually accessed by the
+/// filter expression.
+///
+/// For regular (non-struct) columns, the full field type is used.
+/// For struct columns accessed via `get_field`, a pruned struct type is created
Review Comment:
Okay interesting. I would have thought that `ProjectionMask::leaves` returned a flat column, not a nested struct. Thanks for explaining 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2934150592
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
Understand. But worth following up on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2926161724
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -519,6 +562,149 @@ fn leaf_indices_for_roots(
.collect()
}
+/// Resolves struct field access to specific Parquet leaf column indices
+///
+/// For every `StructFieldAccess`, finds the leaf columns in the Parquet schema
+/// whose path matches the struct root name + field path. This avoids reading all
+/// leaves of a struct when only specific fields are needed
+fn resolve_struct_field_leaves(
+ accesses: &[StructFieldAccess],
+ file_schema: &Schema,
+ schema_descr: &SchemaDescriptor,
+) -> Vec<usize> {
+ let mut leaf_indices = Vec::new();
+
+ for access in accesses {
+ let root_name = file_schema.field(access.root_index).name();
+ let prefix = std::iter::once(root_name.as_str())
+ .chain(access.field_path.iter().map(|p| p.as_str()))
+ .collect::<Vec<_>>();
+
+ for leaf_idx in 0..schema_descr.num_columns() {
+ let col = schema_descr.column(leaf_idx);
+ let col_path = col.path().parts();
+
+ // A leaf matches if its path starts with our prefix.
+ // e.g., prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+
+ // a leaf matches iff its path starts with our prefix
+ // for example: prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+ let leaf_matches_path = col_path.len() >= prefix.len()
+ && col_path.iter().zip(prefix.iter()).all(|(a, b)| a == b);
+
+ if leaf_matches_path {
+ leaf_indices.push(leaf_idx);
+ }
+ }
+ }
+
+ leaf_indices
+}
+
+/// Builds a filter schema that includes only the fields actually accessed by the
+/// filter expression.
+///
+/// For regular (non-struct) columns, the full field type is used.
+/// For struct columns accessed via `get_field`, a pruned struct type is created
Review Comment:
The constraint is what arrow's parquet reader produces. `ProjectionMask::leaves` still returns a nested `StructArray` with only the projected fields, not flat top-level cols. So `Column("s.f")` wouldn't match the reader output without an extra flattening step in `ArrowPredicate::evaluate`
The pruned struct approach keeps the schema in sync with what the reader naturally returns. We'd need to invent a way to get flat columns from the arrow parquet reader to make that work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914274361
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
To my knowledge, no.
`leaf_indices` is only used to build a `ProjectionMask::leaves`
https://github.com/apache/datafusion/blob/5af7361a987f3441aa9718c35ef5381f480a9c94/datafusion/datasource-parquet/src/row_filter.rs#L144-L149
[`ProjectionMask` does not care about order](https://arrow.apache.org/rust/src/parquet/arrow/mod.rs.html#291-302). It builds a boolean mask of size `vec![false; num_columns]` and sets via `mask[leaf_idx] = true`.
Other call sites that use `leaf_indices` isn't considering order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2926036684
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -478,8 +505,24 @@ impl TreeNodeVisitor<'_> for PushdownChecker<'_> {
struct PushdownColumns {
/// Sorted, unique column indices into the file schema required to evaluate
/// the filter expression. Must be in ascending order for correct schema
- /// projection matching.
+ /// projection matching. Does not include struct columns accessed via `get_field`.
required_columns: Vec<usize>,
+ /// Struct field accesses via `get_field`. Each entry records the root struct
+ /// column index and the field path being accessed.
+ struct_field_accesses: Vec<StructFieldAccess>,
+}
+
+/// Records a struct field access via `get_field(struct_col, 'field1', 'field2', ...)`.
+///
+/// This allows the row filter to project only the specific Parquet leaf columns
+/// needed by the filter, rather than all leaves of the struct.
+#[derive(Debug, Clone)]
+struct StructFieldAccess {
+ /// Arrow root column index of the struct in the file schema.
+ root_index: usize,
+ /// Field names forming the path into the struct.
+ /// e.g., `["value"]` for `s['value']`, `["outer", "inner"]` for `s['outer']['inner']`.
Review Comment:
We do support it! Here's a test that repros: https://github.com/apache/datafusion/pull/20854/changes/44a02f3ed91d042e755fec5b0267d34238646aed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2927130524
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -519,6 +562,149 @@ fn leaf_indices_for_roots(
.collect()
}
+/// Resolves struct field access to specific Parquet leaf column indices
+///
+/// For every `StructFieldAccess`, finds the leaf columns in the Parquet schema
+/// whose path matches the struct root name + field path. This avoids reading all
+/// leaves of a struct when only specific fields are needed
+fn resolve_struct_field_leaves(
+ accesses: &[StructFieldAccess],
+ file_schema: &Schema,
+ schema_descr: &SchemaDescriptor,
+) -> Vec<usize> {
Review Comment:
Think this makes a lot of sense. I've charted a big picture idea of what this refactor will look like: https://github.com/apache/datafusion/pull/20913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Project only accessed struct leaves in Parquet row filter pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #20854:
URL: https://github.com/apache/datafusion/pull/20854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2932084553
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
Ah makes sense. Fwiw, this was pre-existing to my PR, I'll choose to leave it untouched for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914333467
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
`leaf_indices` and `root_indices` serve different purposes. leaf indices become the ProjectionMask, telling the parquet decoder which physical leaf columns to read from disk. root indicies (+ struct field accesses) become the filter schema, telling Arrow what schema to use when reconstructing the record batch
Arrow just takes whatever decoded leaves are available and assembles them into the schema it was given. So suppose you had leaf_indices=[2] with root_indices[1]. The masks says decode leaf 2 and the schema says give me struct column 1, pruned to just this specific field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2925996261
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
This can happen during schema evolution (a column that was added after the file was written) or partition columns that are projected onto the scan but don't exist physically on disk
In either case, we can't push the filter down because the decoder has no data for this column
I'll add a comment explaining this or do it in a follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2919587066
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
I know this is pre existing, but what is this code path supposed to catch? Under what circumstances would there be a column that doesn't exist in the file schema at this point, and why is it a "projected" column?
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -478,8 +505,24 @@ impl TreeNodeVisitor<'_> for PushdownChecker<'_> {
struct PushdownColumns {
/// Sorted, unique column indices into the file schema required to evaluate
/// the filter expression. Must be in ascending order for correct schema
- /// projection matching.
+ /// projection matching. Does not include struct columns accessed via `get_field`.
required_columns: Vec<usize>,
+ /// Struct field accesses via `get_field`. Each entry records the root struct
+ /// column index and the field path being accessed.
+ struct_field_accesses: Vec<StructFieldAccess>,
+}
+
+/// Records a struct field access via `get_field(struct_col, 'field1', 'field2', ...)`.
+///
+/// This allows the row filter to project only the specific Parquet leaf columns
+/// needed by the filter, rather than all leaves of the struct.
+#[derive(Debug, Clone)]
+struct StructFieldAccess {
+ /// Arrow root column index of the struct in the file schema.
+ root_index: usize,
+ /// Field names forming the path into the struct.
+ /// e.g., `["value"]` for `s['value']`, `["outer", "inner"]` for `s['outer']['inner']`.
Review Comment:
I assume we don't support stuff like `array_has_any(get_field(s, 'items'), 5)`?
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -519,6 +562,149 @@ fn leaf_indices_for_roots(
.collect()
}
+/// Resolves struct field access to specific Parquet leaf column indices
+///
+/// For every `StructFieldAccess`, finds the leaf columns in the Parquet schema
+/// whose path matches the struct root name + field path. This avoids reading all
+/// leaves of a struct when only specific fields are needed
+fn resolve_struct_field_leaves(
+ accesses: &[StructFieldAccess],
+ file_schema: &Schema,
+ schema_descr: &SchemaDescriptor,
+) -> Vec<usize> {
Review Comment:
Let's make sure we share this logic for projections.
More generally: there should be a single place where there is a function along the lines of:
```rust
fn build_parquet_read_plan(expr: &Arc<dyn PhysicalExpr>) -> ParquetReadPlan {
...
}
struct ParquetReadPlan {
// leaf projections
projection_mask: ProjectionMask,
// the schema to read back with
schema: SchemaRef,
// the transformed expression (do we need this?)
expr: Arc<dyn PhysicalExpr>
}
```
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -519,6 +562,149 @@ fn leaf_indices_for_roots(
.collect()
}
+/// Resolves struct field access to specific Parquet leaf column indices
+///
+/// For every `StructFieldAccess`, finds the leaf columns in the Parquet schema
+/// whose path matches the struct root name + field path. This avoids reading all
+/// leaves of a struct when only specific fields are needed
+fn resolve_struct_field_leaves(
+ accesses: &[StructFieldAccess],
+ file_schema: &Schema,
+ schema_descr: &SchemaDescriptor,
+) -> Vec<usize> {
+ let mut leaf_indices = Vec::new();
+
+ for access in accesses {
+ let root_name = file_schema.field(access.root_index).name();
+ let prefix = std::iter::once(root_name.as_str())
+ .chain(access.field_path.iter().map(|p| p.as_str()))
+ .collect::<Vec<_>>();
+
+ for leaf_idx in 0..schema_descr.num_columns() {
+ let col = schema_descr.column(leaf_idx);
+ let col_path = col.path().parts();
+
+ // A leaf matches if its path starts with our prefix.
+ // e.g., prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+
+ // a leaf matches iff its path starts with our prefix
+ // for example: prefix=["s", "value"] matches leaf path ["s", "value"]
+ // prefix=["s", "outer"] matches ["s", "outer", "inner"]
+ let leaf_matches_path = col_path.len() >= prefix.len()
+ && col_path.iter().zip(prefix.iter()).all(|(a, b)| a == b);
+
+ if leaf_matches_path {
+ leaf_indices.push(leaf_idx);
+ }
+ }
+ }
+
+ leaf_indices
+}
+
+/// Builds a filter schema that includes only the fields actually accessed by the
+/// filter expression.
+///
+/// For regular (non-struct) columns, the full field type is used.
+/// For struct columns accessed via `get_field`, a pruned struct type is created
Review Comment:
Instead of a pruned struct type, why not transform the expression from `get_field(s, 'f')` -> Alias(Column("s.f", 123), "get_field(s, 'f')")` or something like that? Then we don't need to manipulate the data (assemble a struct with pruned fields)
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -449,7 +466,17 @@ impl TreeNodeVisitor<'_> for PushdownChecker<'_> {
{
let return_type = func.return_type();
if !DataType::is_nested(return_type) {
- if let Some(recursion) = self.check_struct_field_column(column.name())
+ let field_path = args[1..]
+ .iter()
+ .filter_map(|arg| {
+ arg.as_any().downcast_ref::<Literal>().and_then(|lit| {
+ lit.value().try_as_str().flatten().map(|s| s.to_string())
+ })
+ })
+ .collect();
Review Comment:
Safer behavior would be to fall back to reading the entire struct root and passing that to the `get_field` UDF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914247807
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
Will this cause errors if the sorting is different from the `root_indices`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2914316524
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -251,15 +251,26 @@ impl FilterCandidateBuilder {
return Ok(None);
};
+ let schema_descr = metadata.file_metadata().schema_descr();
let root_indices: Vec<_> =
required_columns.required_columns.into_iter().collect();
- let leaf_indices = leaf_indices_for_roots(
- &root_indices,
- metadata.file_metadata().schema_descr(),
+ let mut leaf_indices = leaf_indices_for_roots(&root_indices, schema_descr);
+
+ let struct_leaf_indices = resolve_struct_field_leaves(
+ &required_columns.struct_field_accesses,
+ &self.file_schema,
+ schema_descr,
);
+ leaf_indices.extend_from_slice(&struct_leaf_indices);
+ leaf_indices.sort_unstable();
Review Comment:
Ok, then is that change needed? I.e, the sort and dedup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Teach Datafusion to project only accessed struct leaves in row filter pushdown [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2925998379
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -449,7 +466,17 @@ impl TreeNodeVisitor<'_> for PushdownChecker<'_> {
{
let return_type = func.return_type();
if !DataType::is_nested(return_type) {
- if let Some(recursion) = self.check_struct_field_column(column.name())
+ let field_path = args[1..]
+ .iter()
+ .filter_map(|arg| {
+ arg.as_any().downcast_ref::<Literal>().and_then(|lit| {
+ lit.value().try_as_str().flatten().map(|s| s.to_string())
+ })
+ })
+ .collect();
Review Comment:
That's fair 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org