You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/26 15:17:04 UTC

[PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]

alamb opened a new pull request, #14898:
URL: https://github.com/apache/datafusion/pull/14898

   ## Which issue does this PR close?
   
   - Related to https://github.com/delta-io/delta-rs/pull/3261
   - Related to https://github.com/apache/datafusion/issues/14123
   - Follow on to https://github.com/apache/datafusion/pull/14224
   
   
   ## Rationale for this change
   
   While working on upgrading DataFusion 46 in several projects, I found the name `source` super confusing
   
   The issue is there are two similarly named traits,  `FileSource` and `DataSource` but several functions / fields are named just `source` so it is not clear which they refer to 
   
   This was especially confusing as `DataSourceExec` has a `DataSource` which is a `FileScanConfig` not the `FileSOurce` (which is a``ParquetSource` 🤯 
   
   
   ## What changes are included in this PR?
   
   1. rename the fields and accessors to be explicitly `data_source` or `file_source` to avoid the confusion
   
   The documentation on how all these traits are related to each other is somewhat sparse. I am hoping to make a PR to improve it too
   
   ## Are these changes tested?
   By CI
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   While this is an API change, the releveant APIs were introduced in https://github.com/apache/datafusion/pull/14224 and thus not yet released, so I don't think this is explicitly a breaking change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada merged PR #14898:
URL: https://github.com/apache/datafusion/pull/14898


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Rename `DataSource` and `FileSource` fields for consistency [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14898:
URL: https://github.com/apache/datafusion/pull/14898#discussion_r1971829934


##########
datafusion/datasource/src/source.rs:
##########
@@ -139,51 +142,50 @@ impl ExecutionPlan for DataSourceExec {
         partition: usize,
         context: Arc<TaskContext>,
     ) -> datafusion_common::Result<SendableRecordBatchStream> {
-        self.source.open(partition, context)
+        self.data_source.open(partition, context)
     }
 
     fn metrics(&self) -> Option<MetricsSet> {
-        Some(self.source.metrics().clone_inner())
+        Some(self.data_source.metrics().clone_inner())
     }
 
     fn statistics(&self) -> datafusion_common::Result<Statistics> {
-        self.source.statistics()
+        self.data_source.statistics()
     }
 
     fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>> {
-        let mut source = Arc::clone(&self.source);
-        source = source.with_fetch(limit)?;
+        let data_source = self.data_source.with_fetch(limit)?;
         let cache = self.cache.clone();
 
-        Some(Arc::new(Self { source, cache }))
+        Some(Arc::new(Self { data_source, cache }))
     }
 
     fn fetch(&self) -> Option<usize> {
-        self.source.fetch()
+        self.data_source.fetch()
     }
 
     fn try_swapping_with_projection(
         &self,
         projection: &ProjectionExec,
     ) -> datafusion_common::Result<Option<Arc<dyn ExecutionPlan>>> {
-        self.source.try_swapping_with_projection(projection)
+        self.data_source.try_swapping_with_projection(projection)
     }
 }
 
 impl DataSourceExec {
-    pub fn new(source: Arc<dyn DataSource>) -> Self {
-        let cache = Self::compute_properties(Arc::clone(&source));
-        Self { source, cache }
+    pub fn new(data_source: Arc<dyn DataSource>) -> Self {
+        let cache = Self::compute_properties(Arc::clone(&data_source));
+        Self { data_source, cache }
     }
 
     /// Return the source object
-    pub fn source(&self) -> &Arc<dyn DataSource> {
-        &self.source
+    pub fn data_source(&self) -> &Arc<dyn DataSource> {

Review Comment:
   Here is the actual API change -- the rest of the PR is making local variable names consistent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org