You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/26 15:17:04 UTC
[PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]
alamb opened a new pull request, #14898:
URL: https://github.com/apache/datafusion/pull/14898
## Which issue does this PR close?
- Related to https://github.com/delta-io/delta-rs/pull/3261
- Related to https://github.com/apache/datafusion/issues/14123
- Follow on to https://github.com/apache/datafusion/pull/14224
## Rationale for this change
While working on upgrading DataFusion 46 in several projects, I found the name `source` super confusing
The issue is there are two similarly named traits, `FileSource` and `DataSource` but several functions / fields are named just `source` so it is not clear which they refer to
This was especially confusing as `DataSourceExec` has a `DataSource` which is a `FileScanConfig` not the `FileSOurce` (which is a``ParquetSource` 🤯
## What changes are included in this PR?
1. rename the fields and accessors to be explicitly `data_source` or `file_source` to avoid the confusion
The documentation on how all these traits are related to each other is somewhat sparse. I am hoping to make a PR to improve it too
## Are these changes tested?
By CI
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
While this is an API change, the releveant APIs were introduced in https://github.com/apache/datafusion/pull/14224 and thus not yet released, so I don't think this is explicitly a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada merged PR #14898:
URL: https://github.com/apache/datafusion/pull/14898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14898:
URL: https://github.com/apache/datafusion/pull/14898#discussion_r1971829934
##########
datafusion/datasource/src/source.rs:
##########
@@ -139,51 +142,50 @@ impl ExecutionPlan for DataSourceExec {
partition: usize,
context: Arc<TaskContext>,
) -> datafusion_common::Result<SendableRecordBatchStream> {
- self.source.open(partition, context)
+ self.data_source.open(partition, context)
}
fn metrics(&self) -> Option<MetricsSet> {
- Some(self.source.metrics().clone_inner())
+ Some(self.data_source.metrics().clone_inner())
}
fn statistics(&self) -> datafusion_common::Result<Statistics> {
- self.source.statistics()
+ self.data_source.statistics()
}
fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>> {
- let mut source = Arc::clone(&self.source);
- source = source.with_fetch(limit)?;
+ let data_source = self.data_source.with_fetch(limit)?;
let cache = self.cache.clone();
- Some(Arc::new(Self { source, cache }))
+ Some(Arc::new(Self { data_source, cache }))
}
fn fetch(&self) -> Option<usize> {
- self.source.fetch()
+ self.data_source.fetch()
}
fn try_swapping_with_projection(
&self,
projection: &ProjectionExec,
) -> datafusion_common::Result<Option<Arc<dyn ExecutionPlan>>> {
- self.source.try_swapping_with_projection(projection)
+ self.data_source.try_swapping_with_projection(projection)
}
}
impl DataSourceExec {
- pub fn new(source: Arc<dyn DataSource>) -> Self {
- let cache = Self::compute_properties(Arc::clone(&source));
- Self { source, cache }
+ pub fn new(data_source: Arc<dyn DataSource>) -> Self {
+ let cache = Self::compute_properties(Arc::clone(&data_source));
+ Self { data_source, cache }
}
/// Return the source object
- pub fn source(&self) -> &Arc<dyn DataSource> {
- &self.source
+ pub fn data_source(&self) -> &Arc<dyn DataSource> {
Review Comment:
Here is the actual API change -- the rest of the PR is making local variable names consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org