You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/28 15:36:41 UTC
[I] `FileSource` and `DataSource` traits require deep copies [datafusion]
alamb opened a new issue, #14939:
URL: https://github.com/apache/datafusion/issues/14939
### Is your feature request related to a problem or challenge?
While working on various upgrade PRs and preparing for the DataFusion 46 release, I have noticed something I would like to change before we release
The `FileSource` and `DataSource` traits were introduced in the datasource refactor
- https://github.com/apache/datafusion/pull/14224
They have APIs to update the underlying source in a few ways, but the APIs require cloning. For example, `FileSource` looks like this:
```rust
/// Common behaviors that every file format needs to implement.
///
/// See initialization examples on `ParquetSource`, `CsvSource`
pub trait FileSource: Send + Sync {
...
/// Initialize new type with batch size configuration
fn with_batch_size(&self, batch_size: usize) -> Arc<dyn FileSource>;
...
}
```
The only way to implement `with_batch_size` is to (deep) clone the object
```rust
fn with_batch_size(&self, batch_size: usize) -> Arc<dyn FileSource> {
let mut conf = self.clone();
conf.batch_size = Some(batch_size);
Arc::new(conf)
}
```
https://github.com/apache/datafusion/blob/1ae06a497e7c6b117c211c52b33445c2063b9921/datafusion/core/src/datasource/physical_plan/csv.rs#L584-L588
### Describe the solution you'd like
I would like to avoid having to deep clone the object
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `FileSource` and `DataSource` traits require deep copies [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14939:
URL: https://github.com/apache/datafusion/issues/14939#issuecomment-2691065652
Maybe we can try a mutable API instead of a builder style API 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `FileSource` and `DataSource` traits require deep copies [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14939:
URL: https://github.com/apache/datafusion/issues/14939#issuecomment-2694137152
> Should we simply avoid putting these config methods in the trait, but just as each implementation's method? 🤔
That certainly sounds like a good thing to try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `FileSource` and `DataSource` traits require deep copies [datafusion]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on issue #14939:
URL: https://github.com/apache/datafusion/issues/14939#issuecomment-2692703894
Should we simply avoid putting these config methods in the trait, but just as each implementation's method? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `FileSource` and `DataSource` traits require deep copies [datafusion]
Posted by "Standing-Man (via GitHub)" <gi...@apache.org>.
Standing-Man commented on issue #14939:
URL: https://github.com/apache/datafusion/issues/14939#issuecomment-2691760065
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org