You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonathanc-n (via GitHub)" <gi...@apache.org> on 2024/11/29 08:47:05 UTC
[PR] feat: support filter pushdown for datafusion [hudi-rs]
jonathanc-n opened a new pull request, #203:
URL: https://github.com/apache/hudi-rs/pull/203
## Description
<!--- Describe your changes in detail -->
Added `Expr` to `PartitionFilter` conversion to pass in filters. Datafusion will pass down all filters for now using `supports_filters_pushdown` and will filter after partition filters.
<!--- If it fixes an open issue, please link to the issue here. -->
Closes #160.
## Steps forward
I noticed the pr was getting a bit big to be reviewed all at once, here are some things that will be worked on afterwards:
- Support more operators
- Support more Datafusion Expressions
- Enhance the python implementation for filter conversion
## How are the changes test-covered
Added unit tests for all added functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1866341546
##########
python/src/internal.rs:
##########
@@ -163,10 +164,13 @@ impl HudiTable {
filters: Option<Vec<(String, String, String)>>,
Review Comment:
sure, we can leave the python api alignment part to a separate PR. But i think a generic filter struct is still needed for rust api to take in filters from datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1865244265
##########
python/src/internal.rs:
##########
@@ -163,10 +164,13 @@ impl HudiTable {
filters: Option<Vec<(String, String, String)>>,
Review Comment:
Should this be done in another issue? (introducing the generic struct. one of your reviews above mention as well)
##########
python/src/internal.rs:
##########
@@ -163,10 +164,13 @@ impl HudiTable {
filters: Option<Vec<(String, String, String)>>,
Review Comment:
Should this be done in another pr? (introducing the generic struct. one of your reviews above mention as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #203:
URL: https://github.com/apache/hudi-rs/pull/203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#issuecomment-2524967096
> @xushiyan Should we add some docs to see how to run codecov locally? It takes a lot of time out of the review process
Yup having some command in `Makefile` to general html report locally would be beneficial. Codecov aggregates both rust and python, for local mode, not sure how easy to configure that, may start with separate report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#issuecomment-2524959687
@xushiyan Should we add some docs to see how to run codecov locally? It takes a lot of time out of the review process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#issuecomment-2508370026
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/203?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `70.99237%` with `38 lines` in your changes missing coverage. Please review.
> Project coverage is 89.03%. Comparing base [(`a2738da`)](https://app.codecov.io/gh/apache/hudi-rs/commit/a2738da6c15378cae1774a917957a404eb17b242?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`6b72ac1`)](https://app.codecov.io/gh/apache/hudi-rs/commit/6b72ac1e861ef1fbb218a6722219753317152928?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/203?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/datafusion/src/utils/exprs\_to\_filter.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&filepath=crates%2Fdatafusion%2Fsrc%2Futils%2Fexprs_to_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2RhdGFmdXNpb24vc3JjL3V0aWxzL2V4cHJzX3RvX2ZpbHRlci5ycw==) | 72.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python/src/internal.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&filepath=python%2Fsrc%2Finternal.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHl0aG9uL3NyYy9pbnRlcm5hbC5ycw==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/exprs/mod.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fexprs%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2V4cHJzL21vZC5ycw==) | 46.66% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/datafusion/src/lib.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&filepath=crates%2Fdatafusion%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2RhdGFmdXNpb24vc3JjL2xpYi5ycw==) | 76.92% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #203 +/- ##
==========================================
- Coverage 91.78% 89.03% -2.76%
==========================================
Files 20 24 +4
Lines 962 1067 +105
==========================================
+ Hits 883 950 +67
- Misses 79 117 +38
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/203?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1874337378
##########
crates/core/src/table/mod.rs:
##########
@@ -194,7 +195,7 @@ impl Table {
pub async fn get_file_slices_splits(
&self,
n: usize,
- filters: &[(&str, &str, &str)],
+ filters: &[PartitionFilter],
Review Comment:
usually just mark a function`#[deprecated]`, and create the replacing function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#issuecomment-2518891824
fixed conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1865199382
##########
crates/core/src/table/partition.rs:
##########
@@ -40,14 +39,11 @@ pub struct PartitionPruner {
impl PartitionPruner {
pub fn new(
- and_filters: &[(&str, &str, &str)],
+ and_filters: &[PartitionFilter],
partition_schema: &Schema,
hudi_configs: &HudiConfigs,
) -> Result<Self> {
- let and_filters = and_filters
- .iter()
- .map(|filter| PartitionFilter::try_from((*filter, partition_schema)))
- .collect::<Result<Vec<PartitionFilter>>>()?;
+ let and_filters = and_filters.to_vec();
Review Comment:
partition pruner should be one to decide which filter is a valid "partition filter" by checking against the partition schema. Other filters will be ignored here (but will be used by storage layer push down logic which we have not implemented yet). so to summarize the high-level flow is like:
- Hudi filters are constructed from datafusion exprs and pass to the table APIs
- PartitionPruner is then constructed, which could be empty in case no filter is applied to partition fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1866984489
##########
crates/core/src/table/mod.rs:
##########
@@ -194,7 +195,7 @@ impl Table {
pub async fn get_file_slices_splits(
&self,
n: usize,
- filters: &[(&str, &str, &str)],
+ filters: &[PartitionFilter],
Review Comment:
I made the fixes but I was a little confused how to deprecate this. Should I create another similar function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#issuecomment-2525293659
I'm going to take a pass by today and fix wherever necessary, and then merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1865206725
##########
crates/datafusion/src/lib.rs:
##########
@@ -176,6 +215,22 @@ impl TableProvider for HudiDataSource {
Ok(exec_builder.build_arc())
}
+
+ fn supports_filters_pushdown(
+ &self,
+ filters: &[&Expr],
+ ) -> Result<Vec<TableProviderFilterPushDown>> {
+ filters
+ .iter()
+ .map(|expr| {
+ if self.can_push_down(expr) {
+ Ok(TableProviderFilterPushDown::Exact)
Review Comment:
since we only do partition pruning, we can't guarantee all filters used, so it should be `Inexact`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support filter pushdown for datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #203:
URL: https://github.com/apache/hudi-rs/pull/203#discussion_r1865174599
##########
crates/core/src/table/mod.rs:
##########
@@ -194,7 +195,7 @@ impl Table {
pub async fn get_file_slices_splits(
&self,
n: usize,
- filters: &[(&str, &str, &str)],
+ filters: &[PartitionFilter],
Review Comment:
we should provide a generic struct for constructing filter, and not limited to PartitionFilter. And let's mark the existing api deprecated in the upcoming release and only remove in a future release
##########
crates/core/src/exprs/mod.rs:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+pub mod filter;
+
+use anyhow::{anyhow, Error};
+use std::cmp::PartialEq;
+use std::fmt::{Display, Formatter, Result as FmtResult};
+use std::str::FromStr;
+
+pub use filter::*;
+
+/// An operator that represents a comparison operation used in a partition filter expression.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum HudiOperator {
Review Comment:
```suggestion
pub enum ExprOperator {
```
Let's make the name more precise. Plus, within `hudi-core`, we keep a convention to omit "Hudi" prefix since these are all for Hudi implicitly. When imported names have conflicts, we give the imported aliases.
##########
crates/core/src/table/fs_view.rs:
##########
@@ -296,8 +306,16 @@ mod tests {
.await
.unwrap();
let partition_schema = hudi_table.get_partition_schema().await.unwrap();
+
+ let schema = create_test_schema();
+ let filter_lt_20 = PartitionFilter::try_from((("byteField", "<", "20"), &schema))
+ .map_err(|e| anyhow!("Failed to create PartitionFilter: {}", e))
+ .unwrap();
+ let filter_eq_300 = PartitionFilter::try_from((("shortField", "=", "300"), &schema))
+ .map_err(|e| anyhow!("Failed to create PartitionFilter: {}", e))
Review Comment:
these 2 `map_err` are not necessary - they won't fail, just unwrap them. We're removing anyhow in another PR, removing will reduce its footprint.
##########
crates/datafusion/src/utils/exprs_to_filter.rs:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use arrow_array::{Array, Scalar};
+use arrow_schema::SchemaRef;
+use datafusion::logical_expr::Operator;
+use datafusion_expr::{BinaryExpr, Expr};
+use hudi_core::exprs::{HudiOperator, PartitionFilter};
+use std::sync::Arc;
+
+// TODO: Handle other Datafusion `Expr`
+
+/// Converts a slice of DataFusion expressions (`Expr`) into a vector of `PartitionFilter`.
+/// Returns `Some(Vec<PartitionFilter>)` if at least one filter is successfully converted,
+/// otherwise returns `None`.
+pub fn convert_exprs_to_filter(
+ filters: &[Expr],
+ partition_schema: &SchemaRef,
+) -> Vec<PartitionFilter> {
+ let mut partition_filters = Vec::new();
+
+ for expr in filters {
+ match expr {
+ Expr::BinaryExpr(binary_expr) => {
+ if let Some(partition_filter) = convert_binary_expr(binary_expr, partition_schema) {
+ partition_filters.push(partition_filter);
+ }
+ }
+ Expr::Not(not_expr) => {
+ // Handle NOT expressions
+ if let Some(partition_filter) = convert_not_expr(not_expr, partition_schema) {
+ partition_filters.push(partition_filter);
+ }
+ }
+ _ => {
+ continue;
+ }
+ }
+ }
+
+ partition_filters
+}
+
+/// Converts a binary expression (`Expr::BinaryExpr`) into a `PartitionFilter`.
+fn convert_binary_expr(
+ binary_expr: &BinaryExpr,
+ partition_schema: &SchemaRef,
+) -> Option<PartitionFilter> {
+ // extract the column and literal from the binary expression
+ let (column, literal) = match (&*binary_expr.left, &*binary_expr.right) {
+ (Expr::Column(col), Expr::Literal(lit)) => (col, lit),
+ (Expr::Literal(lit), Expr::Column(col)) => (col, lit),
+ _ => return None,
+ };
+
+ let field = partition_schema
+ .field_with_name(column.name())
+ .unwrap()
Review Comment:
this could panic - can we handle the error with a proper datafusion error type?
##########
crates/core/src/exprs/filter.rs:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::exprs::HudiOperator;
+
+use anyhow::{Context, Result};
+use arrow_array::{ArrayRef, Scalar, StringArray};
+use arrow_cast::{cast_with_options, CastOptions};
+use arrow_schema::{DataType, Field, Schema};
+use std::str::FromStr;
+
+/// A partition filter that represents a filter expression for partition pruning.
+#[derive(Debug, Clone)]
+pub struct PartitionFilter {
Review Comment:
partition filter should be exclusively used by `PartitionPruner`, so it should stay within `partition.rs`. Here in `filter.rs`, we can have a generic model for filter, and get pushed down from datafusion or other engine. Only when constructing a `PartitionPruner` can we decide if the filter should be utilized by the pruner
##########
crates/datafusion/src/utils/exprs_to_filter.rs:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use arrow_array::{Array, Scalar};
+use arrow_schema::SchemaRef;
+use datafusion::logical_expr::Operator;
+use datafusion_expr::{BinaryExpr, Expr};
+use hudi_core::exprs::{HudiOperator, PartitionFilter};
+use std::sync::Arc;
+
+// TODO: Handle other Datafusion `Expr`
+
+/// Converts a slice of DataFusion expressions (`Expr`) into a vector of `PartitionFilter`.
+/// Returns `Some(Vec<PartitionFilter>)` if at least one filter is successfully converted,
+/// otherwise returns `None`.
+pub fn convert_exprs_to_filter(
+ filters: &[Expr],
+ partition_schema: &SchemaRef,
+) -> Vec<PartitionFilter> {
+ let mut partition_filters = Vec::new();
+
+ for expr in filters {
+ match expr {
+ Expr::BinaryExpr(binary_expr) => {
+ if let Some(partition_filter) = convert_binary_expr(binary_expr, partition_schema) {
+ partition_filters.push(partition_filter);
+ }
+ }
+ Expr::Not(not_expr) => {
+ // Handle NOT expressions
+ if let Some(partition_filter) = convert_not_expr(not_expr, partition_schema) {
+ partition_filters.push(partition_filter);
+ }
+ }
+ _ => {
+ continue;
+ }
+ }
+ }
+
+ partition_filters
+}
+
+/// Converts a binary expression (`Expr::BinaryExpr`) into a `PartitionFilter`.
+fn convert_binary_expr(
Review Comment:
instead of helper functions, can we implement `TryFrom` for Hudi's generic filter struct to convert Expr to Hudi filter?
##########
crates/datafusion/src/utils/exprs_to_filter.rs:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use arrow_array::{Array, Scalar};
+use arrow_schema::SchemaRef;
+use datafusion::logical_expr::Operator;
+use datafusion_expr::{BinaryExpr, Expr};
+use hudi_core::exprs::{HudiOperator, PartitionFilter};
+use std::sync::Arc;
+
+// TODO: Handle other Datafusion `Expr`
+
+/// Converts a slice of DataFusion expressions (`Expr`) into a vector of `PartitionFilter`.
+/// Returns `Some(Vec<PartitionFilter>)` if at least one filter is successfully converted,
+/// otherwise returns `None`.
+pub fn convert_exprs_to_filter(
+ filters: &[Expr],
+ partition_schema: &SchemaRef,
Review Comment:
as previous comment, we convert whatever filters we support today (e.g., depend on the operators), and push down generic filters to table API, and then let `PartitionPruner` to decide which one to use for partition pruning. We may not need to check against schema here, as the `Expr` should be valid as being provided by datafusion.
##########
crates/datafusion/src/lib.rs:
##########
@@ -129,10 +167,11 @@ impl TableProvider for HudiDataSource {
) -> Result<Arc<dyn ExecutionPlan>> {
self.table.register_storage(state.runtime_env().clone());
+ // Convert Datafusion `Expr` to `PartitionFilter`
+ let partition_filters = convert_exprs_to_filter(filters, &self.schema());
Review Comment:
`self.schema()` is table schema, not partition schema.
##########
python/src/internal.rs:
##########
@@ -163,10 +164,13 @@ impl HudiTable {
filters: Option<Vec<(String, String, String)>>,
Review Comment:
we should keep python API aligned with rust API. if we introduce a generic filter struct, we would need to bind it over here using pyo3, and then use it in the API
##########
python/src/internal.rs:
##########
@@ -201,14 +206,37 @@ impl HudiTable {
filters: Option<Vec<(String, String, String)>>,
py: Python,
) -> PyResult<PyObject> {
- rt().block_on(
- self.inner
- .read_snapshot(vec_to_slice!(filters.unwrap_or_default())),
- )?
- .to_pyarrow(py)
+ let schema: Schema = rt().block_on(self.inner.get_schema())?;
+ let partition_filters = convert_filters(filters, &schema)?;
+
+ rt().block_on(self.inner.read_snapshot(partition_filters.as_slice()))?
+ .to_pyarrow(py)
}
}
+// Temporary fix
Review Comment:
what does this comment mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org