You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rspears74 (via GitHub)" <gi...@apache.org> on 2024/01/18 15:22:48 UTC
[I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
rspears74 opened a new issue, #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906
### Is your feature request related to a problem or challenge?
As far as I can tell, there is no good way to load a subset of files from a partitioned table. Using `ListingTable` or another `TableProvider` like `DeltaTableProvider` from `deltalake`, I'm able to `read_table`, but this loads the entire table. I can also load a list of parquet files with `read_parquet`, but this doesn't work with partitioned tables if the partitions are not "materialized" columns in the raw parquet. The only way I've found to load partitioned files is by iterating over a list of file paths, and doing the entire `TableProvider`/`read_table` process on each one individually, and `union`ing the results together.
### Describe the solution you'd like
It seems like it would be nice to be able to create a `TableProvider` with a table path, then pass some sort of file "whitelist" in. Maybe a `read_table_files(TableProvider, impl IntoIterator<Item = String>)`.
### Describe alternatives you've considered
As stated above, I've tried reading the files one-by-one and `union`ing results, but it's shockingly inefficient compared to reading all files at once.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1912679634
> And 2: Instead, I am trying to essentially copy the implementation of ListingTable and swap out the FileScanConfig used in it's scan method, but I'
Yeah, the current design of ListingTable isn't really setup at the moment to be super modular to swap out the way it determines files, etc. I think making your own version is probably the way to go.
Perhaps this can offer some inspiration: https://github.com/apache/arrow-datafusion/blob/fc752557204f4b52ab4cb38b5caff99b1b73b902/datafusion/core/tests/parquet/schema_coercion.rs#L62-L75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1900451001
I suspect this is glossing over a lot of details of what delta-rs is doing, e.g. schema coercion, etc... but if you have already identified the parquet files in question, it should just be a case of constructing a https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html with the relevant https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/struct.FileScanConfig.html including any partitioning.
As you are effectively side-stepping the catalog implementation, in this case delta-rs, you likely will need to either get the functionality added in there, or reimplement it yourself somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "rspears74 (via GitHub)" <gi...@apache.org>.
rspears74 commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1904709618
I've been looking into this, and running into two issues: 1: Unsure of how to actually use a `ParquetExec` after I've defined it. And 2: Instead, I am trying to essentially copy the implementation of `ListingTable` and swap out the `FileScanConfig` used it it's `scan` method, but I'm hitting some private mods and methods and having to duplicate a bunch of stuff, and the duplication is getting out of hand. If I instead know how to use a `ParquetExec`, I won't have to deal with issue 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1913246376
> I've figured out that I don't actually _need_ this right now (I can do everything without touching the partition columns in my table), but it still stands as a feature request I'd love to see at some point! Thanks @alamb and @tustvold
Glad to hear you got it to work.
I think splitting the ListingTable out a bit more so it can be customized more would help towards this goal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8906:
URL: https://github.com/apache/datafusion/issues/8906#issuecomment-2585712406
Another approach could be if we support metadata columns here:
- https://github.com/apache/datafusion/pull/14057
Would be to expose the filename as a metadata column from a listing table, and then implement filter pushdown on it;
```sql
select ...
FROM my_listing_table
WHERE filename IN ('foo.parquet', 'bar.parquet')
```
Or something like that 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1899999751
If your query contains a predicate this will be used to prune out partitions from the scan. So of you're partitioning on column `a`, if you specify an equality predicate on that column, it will only read files in that partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "rspears74 (via GitHub)" <gi...@apache.org>.
rspears74 commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1900445141
My use case is incremental processing of an append-only delta table:
1. My source table is updated.
2. I inspect the delta log to see which files were added to the table.
3. Process those files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "rspears74 (via GitHub)" <gi...@apache.org>.
rspears74 commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1901238702
Another important thing is I don't actually need to use delta-rs. If there were some way to use a generic `ListingTable`, that would work as well. `read_parquet` works great, but it doesn't do partition discovery and resolution the way `read_table`/`ListingTable` does. I suppose by constructing the two structs you mentioned, I could implement something like that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No efficient way to load a subset of files from partitioned table [arrow-datafusion]
Posted by "rspears74 (via GitHub)" <gi...@apache.org>.
rspears74 commented on issue #8906:
URL: https://github.com/apache/arrow-datafusion/issues/8906#issuecomment-1912811595
I've figured out that I don't actually _need_ this right now (I can do everything without touching the partition columns in my table), but it still stands as a feature request I'd love to see at some point! Thanks @alamb and @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org