You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/05/30 11:51:01 UTC
[PR] Simplify FileSource / SchemaAdapterFactory API [datafusion]
alamb opened a new pull request, #16214:
URL: https://github.com/apache/datafusion/pull/16214
## Which issue does this PR close?
- Follow on for https://github.com/apache/datafusion/pull/16148
## Rationale for this change
During the review of https://github.com/apache/datafusion/pull/16148 from @kosiew 🙏 @adriangb and I had some suggestions to simplify the API and make upgrading easier. However, it wasn't clear how they would look so I tried them out and it turns out they worked well
## What changes are included in this PR?
This PR simplifies the API:
1. Change `FileForma::with_schema_adapter_factory` to return an Result
1. Provides a default implementation for with_schema_adapter_factory (which avoids having to change `FileFormat` implementations)
2. Removes the `impl_schema_adapter_methods` macro
3. Consolidate the newly added standalone integration tests (as individual tests results in longer build times, as each test ends up as its own binary)
## Are these changes tested?
Yes by CI
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Simplify FileSource / SchemaAdapterFactory API [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #16214:
URL: https://github.com/apache/datafusion/pull/16214#discussion_r2115790804
##########
datafusion/core/tests/integration_tests/schema_adapter_integration_tests.rs:
##########
@@ -258,3 +259,187 @@ fn test_schema_adapter_preservation() {
// Verify the schema adapter factory is present in the file source
assert!(config.source().schema_adapter_factory().is_some());
}
+
Review Comment:
this test is moved without change from `datafusion/core/tests/test_adapter_updated.rs`
##########
datafusion/datasource/src/file.rs:
##########
@@ -126,19 +126,26 @@ pub trait FileSource: Send + Sync {
/// Set optional schema adapter factory.
///
/// [`SchemaAdapterFactory`] allows user to specify how fields from the
- /// file get mapped to that of the table schema. The default implementation
- /// returns the original source.
+ /// file get mapped to that of the table schema. If you implement this
+ /// method, you should also implement [`schema_adapter_factory`].
///
- /// Note: You can implement this method and `schema_adapter_factory`
- /// automatically using the [`crate::impl_schema_adapter_methods`] macro.
+ /// The default implementation returns a not implemented error.
+ ///
+ /// [`schema_adapter_factory`]: Self::schema_adapter_factory
Review Comment:
this signature is changed to return a Result and defaults to not implemented
As @kosiew points out this means that implementors of a `FileSource` will only get an error at runtime, not compile time, but I think that is better than forcing everyone to implement a schema adapter even if they don't need it
##########
datafusion/core/src/datasource/physical_plan/arrow_file.rs:
##########
@@ -99,7 +99,19 @@ impl FileSource for ArrowSource {
"arrow"
}
- impl_schema_adapter_methods!();
+ fn with_schema_adapter_factory(
Review Comment:
instead of the `impl_schema_adapter_methods` macro, I instead replicated the code several places. While there is more duplicated code I think it is clearer because what is happening is now more explicit and doesn't require a macro (even though `impl_schema_adapter_methods` was super well documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Simplify FileSource / SchemaAdapterFactory API [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16214:
URL: https://github.com/apache/datafusion/pull/16214#issuecomment-2939472367
Thank you @xudong963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Simplify FileSource / SchemaAdapterFactory API [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16214:
URL: https://github.com/apache/datafusion/pull/16214#issuecomment-2935683108
@xudong963 would you have a moment to help review this PR? It is something I would like to get in before 48.0.0 is released (as it contains a change to an as yet unreleased API)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Simplify FileSource / SchemaAdapterFactory API [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 merged PR #16214:
URL: https://github.com/apache/datafusion/pull/16214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org