You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "liurenjie1024 (via GitHub)" <gi...@apache.org> on 2024/08/26 05:44:51 UTC
[I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
liurenjie1024 opened a new issue, #585:
URL: https://github.com/apache/iceberg-rust/issues/585
This would help to pruning unnecessary data when doing table scan from datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "a-agmon (via GitHub)" <gi...@apache.org>.
a-agmon commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2310334221
Hi
@sdd , @FANNG1 , @liurenjie1024.
I just want to make sure im in the right direction with this and would be happy to hear your feedback on this.
The main functionality will be implemented in `IcebergTableScan` [here](https://github.com/apache/iceberg-rust/blob/905ebd2f332cd37fc86a3ddf845c335229d0abcd/crates/integrations/datafusion/src/physical_plan/scan.rs#L49).
we can pass the filters `&[Expr]` via the `IcebergTableScan::new` function, and then convert them into Iceberg Predicates. Something like this:
```rust
pub(crate) fn new(table: Table, schema: ArrowSchemaRef, filters: &[Expr]) -> Self {
let plan_properties = Self::compute_properties(schema.clone());
let predicates = Self::convert_to_predicates(filters);
println!("==> filters: {:?}", filters);
Self {
table,
schema,
plan_properties,
predicates,
}
}
```
Then the predicates needs to be applied as followes:
The impl of the `ExecutionPlan::execute()` function, calls `get_batch_stream()`, which in turn calls `table.scan()` [link](https://github.com/apache/iceberg-rust/blob/905ebd2f332cd37fc86a3ddf845c335229d0abcd/crates/integrations/datafusion/src/physical_plan/scan.rs#L128)
In this function, the predicate can be applied to the `TableScanBuilder`
Another important issue is that it seems that according to DataFusion docs, in order to get the filters passed to the `scan()` function, we need to impl the `TableProvider::supports_filters_pushdown` to return the filters we can push down [docs] (https://docs.rs/datafusion/41.0.0/datafusion/catalog/trait.TableProvider.html#method.supports_filters_pushdown)
what do you think? does this make sense?
(as a start I think I will try to implement just BinaryExpressions and see from there)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #585: Convert datafusion table scan filter into iceberg table scan' filter.
URL: https://github.com/apache/iceberg-rust/issues/585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "a-agmon (via GitHub)" <gi...@apache.org>.
a-agmon commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2309626060
Hi @FANNG1 ,
I started looking into this and asked @liurenjie1024 to open the issue.
I will update here and would be grateful for your review and advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "a-agmon (via GitHub)" <gi...@apache.org>.
a-agmon commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2316745652
Hi,
I think this is ready for review (will also ping you on slack).
https://github.com/apache/iceberg-rust/pull/588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2309613156
Hi @FANNG1 Thanks for your interest. @a-agmon also volunteered to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2313941236
Hi @a-agmon
> Then the predicates needs to be applied as followes:
> The impl of the ExecutionPlan::execute() function, calls get_batch_stream(), which in turn calls table.scan() [link](https://github.com/apache/iceberg-rust/blob/905ebd2f332cd37fc86a3ddf845c335229d0abcd/crates/integrations/datafusion/src/physical_plan/scan.rs#L128)
> In this function, the predicate can be applied to the TableScanBuilder
Yes, that's I think how it should work.
> Another important issue is that it seems that according to DataFusion docs, in order to get the filters passed to the scan() function, we need to impl the TableProvider::supports_filters_pushdown to return the filters we can push down [docs] (https://docs.rs/datafusion/41.0.0/datafusion/catalog/trait.TableProvider.html#method.supports_filters_pushdown)
>
> what do you think? does this make sense?
> (as a start I think I will try to implement just BinaryExpressions and see from there)
Yes, it makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
FANNG1 commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2309411296
@liurenjie1024 have you worked on this? if not, I'd like to take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
FANNG1 commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2311450677
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]
Posted by "FANNG1 (via GitHub)" <gi...@apache.org>.
FANNG1 commented on issue #585:
URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2309724665
got it, @a-agmon please go ahead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org