You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/31 15:14:55 UTC
[GitHub] [arrow-datafusion] tustvold opened a new pull request, #2668: Remove ObjectStore from FileScanConfig and ListingTableConfig
tustvold opened a new pull request, #2668:
URL: https://github.com/apache/arrow-datafusion/pull/2668
_Draft as builds on #2665 and #2660_
# Which issue does this PR close?
Part of https://github.com/apache/arrow-datafusion/issues/2489
# Rationale for this change
Manually plumbing the ObjectStore around introduces the potential for inconsistency, and increases code complexity.
# What changes are included in this PR?
Follows on from https://github.com/apache/arrow-datafusion/pull/2578#discussion_r884336279 and removes the ObjectStore from FileScanConfig and ListingTableConfig, instead determining as needed from the session context.
# Are there any user-facing changes?
Yes, this changes the API of FileScanConfig and ListingTableConfig.
# Does this PR break compatibility with Ballista?
Probably...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold merged pull request #2668: Remove ObjectStore from FileScanConfig and ListingTableConfig
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2668:
URL: https://github.com/apache/arrow-datafusion/pull/2668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #2668: Remove ObjectStore from FileScanConfig and ListingTableConfig
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2668:
URL: https://github.com/apache/arrow-datafusion/pull/2668#discussion_r887279457
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -589,8 +576,8 @@ mod tests {
target_partitions: usize,
output_partitioning: usize,
) -> Result<()> {
- let mock_store =
- TestObjectStore::new_arc(&files.iter().map(|f| (*f, 10)).collect::<Vec<_>>());
+ let ctx = SessionContext::new();
+ register_test_store(&ctx, &files.iter().map(|f| (*f, 10)).collect::<Vec<_>>());
Review Comment:
This is cool
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -105,10 +99,12 @@ impl ListingTableConfig {
}
/// Infer `ListingOptions` based on `table_path` suffix.
- pub async fn infer_options(self) -> Result<Self> {
+ pub async fn infer_options(self, ctx: &SessionState) -> Result<Self> {
+ let store = ctx.runtime_env.object_store(&self.table_path)?;
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org