You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "vbarua (via GitHub)" <gi...@apache.org> on 2024/08/29 23:43:24 UTC
[PR] validate Substrait NamedScan schemas (#12223) [datafusion]
vbarua opened a new pull request, #12245:
URL: https://github.com/apache/datafusion/pull/12245
Closes https://github.com/apache/datafusion/issues/12223
## Rationale for this change
When DataFusion consumes a Substrait plan, if the schema it has a for a table differs from that given/expected by the Substrait plan it should reject the plan.
Attempting to execute a plan when DataFusion and Substrait disagree on the schema is unlikely to lead to meaningful results.
## What changes are included in this PR?
This PR:
* Adds a validation to the the Substrait plan consumer for NamedScans that rejects Substrait plans when the schema in Substrait does not match that in DataFusion.
* Updates existing tests that fail because of the above validation.
* Updates the Substrait plan producer to include base schemas for ReadRels, which helps with round trip testing (this also caused test failures with the validation).
* Adds an optional parameter to from_substrait_named_struct to allow the field_qualifier to be set (this also caused test failures with the validation).
* Makes the from_substrait_named_struct public as it is generally useful, and also assists with testing.
## Are these changes tested?
Additional tests were added in [substrait_validations.rs](https://github.com/apache/datafusion/compare/main...vbarua:datafusion:vbarua/substrait/validate-schemas?expand=1#diff-eaf82cbac0fc60b73fe8094e6dd6dbc2a0564542defa760ffdbe979f5693eac8) for the validation functionality.
Existing tests failed because of the validation (correctly) and where updated to test their original functinality.
## Are there any user-facing changes?
The added validation could potentially cause calls to `from_substrait_plan` that worked in prior versions to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1747467167
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
let t = t.into_optimized_plan()?;
extract_projection(t, &read.projection)
Review Comment:
I think project unmangling would be better as a follow-up. Possible as part of https://github.com/apache/datafusion/issues/12347 because supporting remaps is going to add yet another layer of Projects 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745686032
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
Review Comment:
we should maybe add this for the local file reads below as well (I didn't have it in my branch yet as I didn't need it immediately). Somewhat annoyingly it'll cause the rest of the TPCH tests to fail...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745715115
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,90 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
Review Comment:
```suggestion
/// nullable. As such if DataFusion has that field as nullable the plan should be rejected.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745450313
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
+fn ensure_field_compatability(
+ datafusion_field: &Field,
+ substrait_field: &Field,
+) -> Result<()> {
+ if DFSchema::datatype_is_logically_equal(
Review Comment:
nit: for easier readability, I'd reverse both this and the inner `if` with `return`s so that you'd have something like:
```
{
if (! datatype_is_logically_equal) {
return substrait_err!(..)
}
if (!nullability check) {
return substrait_err()
}
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745444635
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
+fn ensure_field_compatability(
+ datafusion_field: &Field,
+ substrait_field: &Field,
+) -> Result<()> {
+ if DFSchema::datatype_is_logically_equal(
+ datafusion_field.data_type(),
+ substrait_field.data_type(),
+ ) {
+ if (datafusion_field.is_nullable() == substrait_field.is_nullable())
+ || (substrait_field.is_nullable() && !datafusion_field.is_nullable())
+ {
+ Ok(())
+ } else {
+ // TODO: from_substrait_struct_type needs to be updated to set the nullability correctly. It defaults to true for now.
+ substrait_err!(
+ "Field '{}' is nullable in the Substrait schema but not nullable in the DataFusion schema.",
Review Comment:
Isn't this the other way around? Nullable in DF but not nullable in Substrait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745671825
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1586,9 +1629,12 @@ fn next_struct_field_name(
}
}
-fn from_substrait_named_struct(
+/// Convert Substrait NamedStruct to DataFusion DFSchemaRef
+pub fn from_substrait_named_struct(
Review Comment:
Weird. Well, I guess it's okay to be public too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745692771
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
+fn ensure_field_compatability(
+ datafusion_field: &Field,
+ substrait_field: &Field,
+) -> Result<()> {
+ if DFSchema::datatype_is_logically_equal(
+ datafusion_field.data_type(),
+ substrait_field.data_type(),
+ ) {
+ if (datafusion_field.is_nullable() == substrait_field.is_nullable())
+ || (substrait_field.is_nullable() && !datafusion_field.is_nullable())
+ {
+ Ok(())
+ } else {
+ // TODO: from_substrait_struct_type needs to be updated to set the nullability correctly. It defaults to true for now.
+ substrait_err!(
+ "Field '{}' is nullable in the Substrait schema but not nullable in the DataFusion schema.",
Review Comment:
Heh, of course I got the check logic right but borked the error message. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745801872
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
Review Comment:
I see what you're saying. The table lookup acts like an implicit check on the qualifier. I'll remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate Substrait NamedScan schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1740400324
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1601,12 +1647,17 @@ fn from_substrait_named_struct(
);
if name_idx != base_schema.names.len() {
return substrait_err!(
- "Names list must match exactly to nested schema, but found {} uses for {} names",
- name_idx,
- base_schema.names.len()
- );
+ "Names list must match exactly to nested schema, but found {} uses for {} names",
+ name_idx,
+ base_schema.names.len()
+ );
+ }
+ let mut df_schema = DFSchema::try_from(Schema::new(fields?))?;
+ match field_qualifier {
+ None => (),
+ Some(fq) => df_schema = df_schema.replace_qualifier(fq),
Review Comment:
Is this needed for something? Maybe anyways rather than taking in an optional parameter, just call the `.replace_qualifier` on the returned schema where it's needed.
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +868,31 @@ pub async fn from_substrait_rel(
}
}
+/// Validates that the given Substrait schema matches the given DataFusion schema
+/// Returns true if the two schemas have the same qualified named fields with the same data types.
+/// Returns false otherwise.
+/// Ignores case when comparing field names
+///
+/// This code is equivalent to [DFSchema::equivalent_names_and_types] except that the field name
+/// checking is case-insensitive
+fn validate_substrait_schema(
+ datafusion_schema: &DFSchema,
+ substrait_schema: &DFSchema,
+) -> bool {
+ if datafusion_schema.fields().len() != substrait_schema.fields().len() {
+ return false;
+ }
+ let datafusion_fields = datafusion_schema.iter();
+ let substrait_fields = substrait_schema.iter();
+ datafusion_fields
+ .zip(substrait_fields)
+ .all(|((q1, f1), (q2, f2))| {
+ q1 == q2
+ && f1.name().to_lowercase() == f2.name().to_lowercase()
+ && f1.data_type().equals_datatype(f2.data_type())
Review Comment:
I think this asserts the nullability of inner fields in things like lists. That's probably too strict, we should accept cases where the DF schema is non-nullable but Substrait schema allows null.
##########
datafusion/substrait/src/logical_plan/producer.rs:
##########
@@ -140,19 +140,13 @@ pub fn to_substrait_rel(
maintain_singular_struct: false,
});
+ let table_schema = scan.source.schema().to_dfschema_ref()?;
+ let base_schema = to_substrait_named_struct(&table_schema, extensions)?;
+
Ok(Box::new(Rel {
rel_type: Some(RelType::Read(Box::new(ReadRel {
common: None,
- base_schema: Some(NamedStruct {
- names: scan
- .source
- .schema()
- .fields()
- .iter()
- .map(|f| f.name().to_owned())
- .collect(),
- r#struct: None,
- }),
+ base_schema: Some(base_schema),
Review Comment:
nice!
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1586,9 +1629,12 @@ fn next_struct_field_name(
}
}
-fn from_substrait_named_struct(
+/// Convert Substrait NamedStruct to DataFusion DFSchemaRef
+pub fn from_substrait_named_struct(
Review Comment:
you can do `pub(crate) fn`
##########
datafusion/substrait/tests/utils.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[cfg(test)]
+pub mod test {
+ use datafusion::catalog_common::TableReference;
+ use datafusion::datasource::empty::EmptyTable;
+ use datafusion::prelude::SessionContext;
+ use datafusion_substrait::extensions::Extensions;
+ use datafusion_substrait::logical_plan::consumer::from_substrait_named_struct;
+ use std::fs::File;
+ use std::io::BufReader;
+ use std::sync::Arc;
+ use substrait::proto::read_rel::{NamedTable, ReadType};
+ use substrait::proto::rel::RelType;
+ use substrait::proto::{Plan, ReadRel, Rel};
+
+ pub fn read_json(path: &str) -> Plan {
+ serde_json::from_reader::<_, Plan>(BufReader::new(
+ File::open(path).expect("file not found"),
+ ))
+ .expect("failed to parse json")
+ }
+
+ pub struct TestSchemaCollector {
+ ctx: SessionContext,
+ }
+
+ impl TestSchemaCollector {
+ fn new() -> Self {
+ TestSchemaCollector {
+ ctx: SessionContext::new(),
+ }
+ }
+
+ pub fn generate_context_from_plan(plan: &Plan) -> SessionContext {
+ let mut schema_collector = Self::new();
+
+ for plan_rel in plan.relations.iter() {
+ match plan_rel.rel_type.as_ref().expect("blah") {
Review Comment:
"blah" ? :)
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +868,31 @@ pub async fn from_substrait_rel(
}
}
+/// Validates that the given Substrait schema matches the given DataFusion schema
Review Comment:
I feel like requiring an exact match is unnecessarily harsh. I'd prefer checking that the Substrait schema is a subset of the DF schema. I could see our use-case for example storing a given Substrait plan and running it again and again over time, while the input dataset may change to add new columns. There's no reason why that shouldn't work, right?
(See also the comment about nulls below)
##########
datafusion/substrait/tests/utils.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[cfg(test)]
+pub mod test {
+ use datafusion::catalog_common::TableReference;
+ use datafusion::datasource::empty::EmptyTable;
+ use datafusion::prelude::SessionContext;
+ use datafusion_substrait::extensions::Extensions;
+ use datafusion_substrait::logical_plan::consumer::from_substrait_named_struct;
+ use std::fs::File;
+ use std::io::BufReader;
+ use std::sync::Arc;
+ use substrait::proto::read_rel::{NamedTable, ReadType};
+ use substrait::proto::rel::RelType;
+ use substrait::proto::{Plan, ReadRel, Rel};
+
+ pub fn read_json(path: &str) -> Plan {
+ serde_json::from_reader::<_, Plan>(BufReader::new(
+ File::open(path).expect("file not found"),
+ ))
+ .expect("failed to parse json")
+ }
+
+ pub struct TestSchemaCollector {
+ ctx: SessionContext,
+ }
+
+ impl TestSchemaCollector {
+ fn new() -> Self {
+ TestSchemaCollector {
+ ctx: SessionContext::new(),
+ }
+ }
+
+ pub fn generate_context_from_plan(plan: &Plan) -> SessionContext {
+ let mut schema_collector = Self::new();
+
+ for plan_rel in plan.relations.iter() {
+ match plan_rel.rel_type.as_ref().expect("blah") {
+ substrait::proto::plan_rel::RelType::Rel(r) => {
+ schema_collector.collect_schemas(r)
+ }
+ substrait::proto::plan_rel::RelType::Root(r) => schema_collector
+ .collect_schemas(
+ r.input.as_ref().expect("RelRoot must set input"),
+ ),
+ }
+ }
+ schema_collector.ctx
+ }
+
+ fn collect_named_table(&mut self, read: &ReadRel, nt: &NamedTable) {
+ let table_reference = match nt.names.len() {
+ 0 => {
+ panic!("No table name found in NamedTable");
+ }
+ 1 => TableReference::Bare {
+ table: nt.names[0].clone().into(),
+ },
+ 2 => TableReference::Partial {
+ schema: nt.names[0].clone().into(),
+ table: nt.names[1].clone().into(),
+ },
+ _ => TableReference::Full {
+ catalog: nt.names[0].clone().into(),
+ schema: nt.names[1].clone().into(),
+ table: nt.names[2].clone().into(),
+ },
+ };
+
+ let substrait_schema = read
+ .base_schema
+ .as_ref()
+ .expect("No base schema found for NamedTable");
+ let empty_extensions = Extensions {
+ functions: Default::default(),
+ types: Default::default(),
+ type_variations: Default::default(),
+ };
+
+ let df_schema = from_substrait_named_struct(
+ substrait_schema,
+ &empty_extensions,
+ Some(table_reference.clone()),
+ )
+ .expect("Unable to generate DataFusion schema from Substrait NamedStruct");
+ let table = EmptyTable::new(df_schema.inner().clone());
+ self.ctx
Review Comment:
might be nicer to just return the name and schema (that'd also better match the name "collect") rather than modify ctx directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate Substrait NamedScan schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1737503917
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1586,9 +1629,12 @@ fn next_struct_field_name(
}
}
-fn from_substrait_named_struct(
+/// Convert Substrait NamedStruct to DataFusion DFSchemaRef
+pub fn from_substrait_named_struct(
Review Comment:
While this is generally useful, I could see a case for keeping this internal only for now. That was actually what I tried to do originally, but I'm new-ish to Rust and couldn't figure out a good way to expose this for use in integration tests only.
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1601,12 +1647,17 @@ fn from_substrait_named_struct(
);
if name_idx != base_schema.names.len() {
return substrait_err!(
- "Names list must match exactly to nested schema, but found {} uses for {} names",
- name_idx,
- base_schema.names.len()
- );
+ "Names list must match exactly to nested schema, but found {} uses for {} names",
+ name_idx,
+ base_schema.names.len()
+ );
+ }
+ let mut df_schema = DFSchema::try_from(Schema::new(fields?))?;
+ match field_qualifier {
+ None => (),
+ Some(fq) => df_schema = df_schema.replace_qualifier(fq),
}
- Ok(DFSchemaRef::new(DFSchema::try_from(Schema::new(fields?))?))
+ Ok(DFSchemaRef::new(df_schema))
Review Comment:
If this is made public, I think it might make sense to return a `DFSchema` instead of a `DFSchemaRef` to make it more flexibly for users, whom can choose to wrap the DFSchema if they need it.
##########
datafusion/substrait/tests/cases/function_test.rs:
##########
@@ -19,40 +19,33 @@
#[cfg(test)]
mod tests {
+ use crate::utils::test::TestSchemaCollector;
use datafusion::common::Result;
- use datafusion::prelude::{CsvReadOptions, SessionContext};
use datafusion_substrait::logical_plan::consumer::from_substrait_plan;
use std::fs::File;
use std::io::BufReader;
use substrait::proto::Plan;
#[tokio::test]
async fn contains_function_test() -> Result<()> {
- let ctx = create_context().await?;
-
let path = "tests/testdata/contains_plan.substrait.json";
- let proto = serde_json::from_reader::<_, Plan>(BufReader::new(
+
+ let proto_plan = serde_json::from_reader::<_, Plan>(BufReader::new(
File::open(path).expect("file not found"),
))
.expect("failed to parse json");
- let plan = from_substrait_plan(&ctx, &proto).await?;
+ let ctx = TestSchemaCollector::generate_context_from_plan(&proto_plan);
+ let plan = from_substrait_plan(&ctx, &proto_plan).await?;
let plan_str = format!("{}", plan);
assert_eq!(
plan_str,
- "Projection: nation.b AS n_name\
- \n Filter: contains(nation.b, Utf8(\"IA\"))\
- \n TableScan: nation projection=[a, b, c, d, e, f]"
+ "Projection: nation.n_name\
+ \n Filter: contains(nation.n_name, Utf8(\"IA\"))\
+ \n TableScan: nation projection=[n_nationkey, n_name, n_regionkey, n_comment]"
Review Comment:
You can see in this change how the DataFusion and Substrait plans had different schemas.
##########
datafusion/substrait/tests/utils.rs:
##########
@@ -0,0 +1,154 @@
+#[cfg(test)]
+pub(crate) mod test {
+ use datafusion::catalog_common::TableReference;
+ use datafusion::datasource::empty::EmptyTable;
+ use datafusion::prelude::SessionContext;
+ use datafusion_substrait::extensions::Extensions;
+ use datafusion_substrait::logical_plan::consumer::from_substrait_named_struct;
+ use std::fs::File;
+ use std::io::BufReader;
+ use std::sync::Arc;
+ use substrait::proto::read_rel::{NamedTable, ReadType};
+ use substrait::proto::rel::RelType;
+ use substrait::proto::{Plan, ReadRel, Rel};
+
+ pub(crate) fn read_json(path: &str) -> Plan {
+ serde_json::from_reader::<_, Plan>(BufReader::new(
+ File::open(path).expect("file not found"),
+ ))
+ .expect("failed to parse json")
+ }
+
+ pub(crate) struct TestSchemaCollector {
+ ctx: SessionContext,
+ }
Review Comment:
This collector is based on a similar bit of tooling in [Isthmus](https://github.com/substrait-io/substrait-java/blob/d6256487f75032e2e8fc803976d916e4a5bc44bd/isthmus/src/main/java/io/substrait/isthmus/SubstraitToCalcite.java#L102-L128), a Substrait library used for integrating with Apache Calcite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745691410
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
+fn ensure_field_compatability(
+ datafusion_field: &Field,
+ substrait_field: &Field,
+) -> Result<()> {
+ if DFSchema::datatype_is_logically_equal(
Review Comment:
Good idea, it ends up much easier to read ✨
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745683938
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,79 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// Substrait schema is a subset of the DataFusion schema.
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
Review Comment:
FWIW I don't think it's super necessary/useful to ensure the qualifiers even in future, presumably if DF doesn't have a table with a matching qualifier (names) the query will fail 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#issuecomment-2334710029
I think this is good by me - @alamb would you (or someone else) be able to do the official review, please? :)
Only note I have is that I think this change makes Substrait consumer case-sensitive wrt column names, which it wasn't before. I don't strictly have opinion on whether that's a good or a bad thing. I think for _my_ usecase it would be fine, but dunno about others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745712603
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
let t = t.into_optimized_plan()?;
extract_projection(t, &read.projection)
Review Comment:
I started wondering if the `ensure_schema_compatability` can now conflict with `extract_projection` - and I think it can, either by failing if DF doesn't optimize the select into a projection, or if DF does, then by overriding the select's projection with the Substrait projection...
I guess a fix would be something like in `extract_projection`, if there is an existing `scan.projection`, then apply `columnIndices` on it first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1744627789
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +868,31 @@ pub async fn from_substrait_rel(
}
}
+/// Validates that the given Substrait schema matches the given DataFusion schema
Review Comment:
Good point, I've made the validation more flexible based on the work in your PR.
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +864,82 @@ pub async fn from_substrait_rel(
}
}
+/// Ensures that the given Substrait schema is compatible with the schema as given by DataFusion
+///
+/// This means:
+/// 1. All fields present in the Substrait schema are present in the DataFusion schema. The
+/// DataFusion schema may have MORE fields, but not the other way around.
+/// 2. All fields are compatible. See [`ensure_field_compatability`] for details
+///
+/// This function returns a DataFrame with fields adjusted if necessary in the event that the
+/// SubstraitSchema is a subset of the DataFusion schema.
+// ```
+fn ensure_schema_compatability(
+ table: DataFrame,
+ substrait_schema: DFSchema,
+) -> Result<DataFrame> {
+ // For now strip the qualifiers from the DataFusion and Substrait schemas to make comparisons
+ // easier as there are issues around qualifier case.
+ // TODO: ensure that qualifiers are the same
+ let df_schema = table.schema().to_owned().strip_qualifiers();
+ if df_schema.logically_equivalent_names_and_types(&substrait_schema) {
+ return Ok(table);
+ }
+ let selected_columns = substrait_schema
+ .strip_qualifiers()
+ .fields()
+ .iter()
+ .map(|substrait_field| {
+ let df_field =
+ df_schema.field_with_unqualified_name(substrait_field.name())?;
+ ensure_field_compatability(df_field, substrait_field)?;
+ Ok(col(format!("\"{}\"", df_field.name())))
+ })
+ .collect::<Result<_>>()?;
+
+ table.select(selected_columns)
+}
+
+///
+/// Ensures that the given Substrait field is compatible with the given DataFusion field
+///
+/// A field is compatible between Substrait and DataFusion if:
+/// 1. They have logically equivalent types.
+/// 2. They have the same nullability OR the Substrait field is nullable and the DataFusion fields
+/// is not nullable.
+///
+/// If a Substrait field is not nullable, the Substrait plan may be built around assuming it is not
+/// nullable. As such if DataFusion has that field as not nullable the plan should be rejected.
+//
+fn ensure_field_compatability(
+ datafusion_field: &Field,
+ substrait_field: &Field,
+) -> Result<()> {
+ if DFSchema::datatype_is_logically_equal(
+ datafusion_field.data_type(),
+ substrait_field.data_type(),
+ ) {
+ if (datafusion_field.is_nullable() == substrait_field.is_nullable())
+ || (substrait_field.is_nullable() && !datafusion_field.is_nullable())
+ {
+ Ok(())
+ } else {
+ // TODO: from_substrait_struct_type needs to be updated to set the nullability correctly. It defaults to true for now.
+ substrait_err!(
+ "Field '{}' is nullable in the Substrait schema but not nullable in the DataFusion schema.",
+ substrait_field.name()
+ )
Review Comment:
Because of what's mentioned in the TODO, this code never fires. I'm opting not to fix that as part of this PR.
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1586,9 +1629,12 @@ fn next_struct_field_name(
}
}
-fn from_substrait_named_struct(
+/// Convert Substrait NamedStruct to DataFusion DFSchemaRef
+pub fn from_substrait_named_struct(
Review Comment:
I tried doing that originally, but ran into failures in the intergration tests which failed to compile with:
```
error[E0603]: function `from_substrait_named_struct` is private
--> datafusion/substrait/tests/utils.rs:24:55
|
24 | use datafusion_substrait::logical_plan::consumer::from_substrait_named_struct;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ private function
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #12245:
URL: https://github.com/apache/datafusion/pull/12245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#issuecomment-2340341437
Thanks again everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1744628350
##########
datafusion/substrait/tests/testdata/tpch/nation.csv:
##########
@@ -1,2 +1,2 @@
-n_nationkey,n_name,n_regionkey,n_comment
+N_NATIONKEY,N_NAME,N_REGIONKEY,N_COMMENT
0,ALGERIA,0, haggle. carefully final deposits detect slyly agai
Review Comment:
@Blizzara this is based on what you did in this PR to fix the tests that fail due to case difference in fields name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1746373645
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
let t = t.into_optimized_plan()?;
extract_projection(t, &read.projection)
Review Comment:
That did indeed cause problems. It triggered an error of `unexpected plan for table` in `extract_projection`.
I added some code for this case in [`d571eb2` (#12245)](https://github.com/apache/datafusion/pull/12245/commits/d571eb2784ff54617483ebe4435bbff47dd6bea4). Is something like this what you had in mind?
I am noticing that the plans generated look a little weird/bad with a lot of redundant projects
```
"Projection: DATA.a, DATA.b\
\n Projection: DATA.a, DATA.b\
\n Projection: DATA.a, DATA.b, DATA.c\
\n TableScan: DATA projection=[b, a, c]"
```
but they are at least correct for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1747405489
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
let t = t.into_optimized_plan()?;
extract_projection(t, &read.projection)
Review Comment:
> Is something like this what you had in mind?
What I had in mind was manipulating the `scan.projection` directly - kinda like it is alreadydone in extract_projection, we could do it that way also for `ensure_schema_compatibility`. That way there wouldn't be additional Projections, and maybe it'd be a bit more efficient if the current setup doesn't push the column-pruning into the scan level (though I'm a bit surprised they don't get optimized anyways).
But I don't think it's necessary - the way you've done it here seems correct, and we (I?) can do the project-mangling as a followup, unless you want to take a stab at it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745798223
##########
datafusion/substrait/tests/utils.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[cfg(test)]
+pub mod test {
+ use datafusion::catalog_common::TableReference;
+ use datafusion::datasource::empty::EmptyTable;
+ use datafusion::prelude::SessionContext;
+ use datafusion_substrait::extensions::Extensions;
+ use datafusion_substrait::logical_plan::consumer::from_substrait_named_struct;
+ use std::fs::File;
+ use std::io::BufReader;
+ use std::sync::Arc;
+ use substrait::proto::read_rel::{NamedTable, ReadType};
+ use substrait::proto::rel::RelType;
+ use substrait::proto::{Plan, ReadRel, Rel};
+
+ pub fn read_json(path: &str) -> Plan {
+ serde_json::from_reader::<_, Plan>(BufReader::new(
+ File::open(path).expect("file not found"),
+ ))
+ .expect("failed to parse json")
+ }
+
+ pub struct TestSchemaCollector {
+ ctx: SessionContext,
+ }
+
+ impl TestSchemaCollector {
+ fn new() -> Self {
+ TestSchemaCollector {
+ ctx: SessionContext::new(),
+ }
+ }
+
+ pub fn generate_context_from_plan(plan: &Plan) -> SessionContext {
+ let mut schema_collector = Self::new();
+
+ for plan_rel in plan.relations.iter() {
+ match plan_rel.rel_type.as_ref().expect("blah") {
+ substrait::proto::plan_rel::RelType::Rel(r) => {
+ schema_collector.collect_schemas(r)
+ }
+ substrait::proto::plan_rel::RelType::Root(r) => schema_collector
+ .collect_schemas(
+ r.input.as_ref().expect("RelRoot must set input"),
+ ),
+ }
+ }
+ schema_collector.ctx
+ }
+
+ fn collect_named_table(&mut self, read: &ReadRel, nt: &NamedTable) {
+ let table_reference = match nt.names.len() {
+ 0 => {
+ panic!("No table name found in NamedTable");
+ }
+ 1 => TableReference::Bare {
+ table: nt.names[0].clone().into(),
+ },
+ 2 => TableReference::Partial {
+ schema: nt.names[0].clone().into(),
+ table: nt.names[1].clone().into(),
+ },
+ _ => TableReference::Full {
+ catalog: nt.names[0].clone().into(),
+ schema: nt.names[1].clone().into(),
+ table: nt.names[2].clone().into(),
+ },
+ };
+
+ let substrait_schema = read
+ .base_schema
+ .as_ref()
+ .expect("No base schema found for NamedTable");
+ let empty_extensions = Extensions {
+ functions: Default::default(),
+ types: Default::default(),
+ type_variations: Default::default(),
+ };
+
+ let df_schema = from_substrait_named_struct(
+ substrait_schema,
+ &empty_extensions,
+ Some(table_reference.clone()),
+ )
+ .expect("Unable to generate DataFusion schema from Substrait NamedStruct");
+ let table = EmptyTable::new(df_schema.inner().clone());
+ self.ctx
Review Comment:
Good idea. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1745800664
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
Review Comment:
Is this worth doing as one big swoop in this PR, or would it make sense to do it as a followup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate and adjust Substrait NamedTable schemas (#12223) [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1747397923
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -657,7 +662,13 @@ pub async fn from_substrait_rel(
table: nt.names[2].clone().into(),
},
};
- let t = ctx.table(table_reference).await?;
+
+ let substrait_schema =
+ from_substrait_named_struct(named_struct, extensions)?
+ .replace_qualifier(table_reference.clone());
+
+ let t = ctx.table(table_reference.clone()).await?;
+ let t = ensure_schema_compatability(t, substrait_schema)?;
Review Comment:
Fine by me to do it separately!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] validate Substrait NamedScan schemas (#12223) [datafusion]
Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12245:
URL: https://github.com/apache/datafusion/pull/12245#discussion_r1737501114
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -850,6 +868,31 @@ pub async fn from_substrait_rel(
}
}
+/// Validates that the given Substrait schema matches the given DataFusion schema
+/// Returns true if the two schemas have the same qualified named fields with the same data types.
+/// Returns false otherwise.
+/// Ignores case when comparing field names
+///
+/// This code is equivalent to [DFSchema::equivalent_names_and_types] except that the field name
+/// checking is case-insensitive
+fn validate_substrait_schema(
+ datafusion_schema: &DFSchema,
+ substrait_schema: &DFSchema,
+) -> bool {
+ if datafusion_schema.fields().len() != substrait_schema.fields().len() {
+ return false;
+ }
+ let datafusion_fields = datafusion_schema.iter();
+ let substrait_fields = substrait_schema.iter();
+ datafusion_fields
+ .zip(substrait_fields)
+ .all(|((q1, f1), (q2, f2))| {
+ q1 == q2
+ && f1.name().to_lowercase() == f2.name().to_lowercase()
Review Comment:
I wasn't sure what the actual requirements around case-sensitivity are in DataFusion. I've opted for a more permissive check here, but this could potentially be configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org