You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "phillipleblanc (via GitHub)" <gi...@apache.org> on 2025/03/12 13:06:16 UTC
[PR] Support metadata columns (location, size, last_modified) in ListingTableProvider [datafusion]
phillipleblanc opened a new pull request, #15181:
URL: https://github.com/apache/datafusion/pull/15181
## Which issue does this PR close?
- Closes #15173
## Rationale for this change
The ListingTableProvider in DataFusion provides an implementation of a TableProvider that organizes a collection of (potentially hive partitioned) files in an object store into a single table.
Similar to how hive partitions are injected into the listing table schema, but they don't actually exist in the physical parquet files - this PR adds the ability to request the ListingTable to inject metadata columns that get their data from the ObjectMeta provided by the object store crate. That allows consumers to opt-in for the requested metadata columns.
## What changes are included in this PR?
I've added a new API on the ListingOptions struct that is passed to a ListingTableConfig which is passed to ListingTable::try_new.
```rust
/// Set metadata columns on [`ListingOptions`] and returns self.
///
/// "metadata columns" are columns that are computed from the `ObjectMeta` of the files from object store.
///
/// Available metadata columns:
/// - `location`: The full path to the object
/// - `last_modified`: The last modified time
/// - `size`: The size in bytes of the object
///
/// For example, given the following files in object store:
///
/// ```text
/// /mnt/nyctaxi/tripdata01.parquet
/// /mnt/nyctaxi/tripdata02.parquet
/// /mnt/nyctaxi/tripdata03.parquet
/// ```
///
/// If the `last_modified` field in the `ObjectMeta` for `tripdata01.parquet` is `2024-01-01 12:00:00`,
/// then the table schema will include a column named `last_modified` with the value `2024-01-01 12:00:00`
/// for all rows read from `tripdata01.parquet`.
///
/// | <other columns> | last_modified |
/// |-----------------|-----------------------|
/// | ... | 2024-01-01 12:00:00 |
/// | ... | 2024-01-02 15:30:00 |
/// | ... | 2024-01-03 09:15:00 |
///
/// # Example
/// ```
/// # use std::sync::Arc;
/// # use datafusion::datasource::{listing::ListingOptions, file_format::parquet::ParquetFormat};
///
/// let listing_options = ListingOptions::new(Arc::new(
/// ParquetFormat::default()
/// ))
/// .with_metadata_cols(vec![MetadataColumn::LastModified]);
///
/// assert_eq!(listing_options.metadata_cols, vec![MetadataColumn::LastModified]);
/// ```
pub fn with_metadata_cols(mut self, metadata_cols: Vec<MetadataColumn>) -> Self {
self.metadata_cols = metadata_cols;
self
}
```
That controls whether the ListingTableProvider will add the metadata columns to the schema, similar to how partition columns are added.
Filters on metadata columns directly can be used to prune out files that don't need to be read - i.e. `SELECT * FROM my_listing_table WHERE last_modified > '2025-03-10'` will only scan files that were modified after '2025-03-10'.
## Are these changes tested?
Yes, I've added tests in several places (including adding tests for functions that I've changed that didn't previously exist).
## Are there any user-facing changes?
The main change is adding the `with_metadata_cols` API on the `ListingOptions` struct. This is not a breaking change, as the current behavior will be to not add any metadata columns unless `with_metadata_cols` is explicitly called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on PR #15181:
URL: https://github.com/apache/datafusion/pull/15181#issuecomment-2856679733
Closing in favor of the approach outlined in #15173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support metadata columns (location, size, last_modified) in ListingTableProvider [datafusion]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on code in PR #15181:
URL: https://github.com/apache/datafusion/pull/15181#discussion_r1991476025
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -417,6 +422,50 @@ impl ListingOptions {
self
}
+ /// Set metadata columns on [`ListingOptions`] and returns self.
+ ///
+ /// "metadata columns" are columns that are computed from the `ObjectMeta` of the files from object store.
+ ///
+ /// Available metadata columns:
+ /// - `location`: The full path to the object
+ /// - `last_modified`: The last modified time
+ /// - `size`: The size in bytes of the object
+ ///
+ /// For example, given the following files in object store:
+ ///
+ /// ```text
+ /// /mnt/nyctaxi/tripdata01.parquet
+ /// /mnt/nyctaxi/tripdata02.parquet
+ /// /mnt/nyctaxi/tripdata03.parquet
+ /// ```
+ ///
+ /// If the `last_modified` field in the `ObjectMeta` for `tripdata01.parquet` is `2024-01-01 12:00:00`,
+ /// then the table schema will include a column named `last_modified` with the value `2024-01-01 12:00:00`
+ /// for all rows read from `tripdata01.parquet`.
+ ///
+ /// | [other columns] | last_modified |
+ /// |-----------------|-----------------------|
+ /// | ... | 2024-01-01 12:00:00 |
+ /// | ... | 2024-01-02 15:30:00 |
+ /// | ... | 2024-01-03 09:15:00 |
+ ///
+ /// # Example
+ /// ```
+ /// # use std::sync::Arc;
+ /// # use datafusion::datasource::{listing::ListingOptions, file_format::parquet::ParquetFormat};
+ ///
+ /// let listing_options = ListingOptions::new(Arc::new(
+ /// ParquetFormat::default()
+ /// ))
+ /// .with_metadata_cols(vec![MetadataColumn::LastModified]);
+ ///
+ /// assert_eq!(listing_options.metadata_cols, vec![MetadataColumn::LastModified]);
+ /// ```
+ pub fn with_metadata_cols(mut self, metadata_cols: Vec<MetadataColumn>) -> Self {
+ self.metadata_cols = metadata_cols;
+ self
+ }
Review Comment:
This is the main API change that consumers would use to enable these columns on the Listing Table. They aren't added by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on PR #15181:
URL: https://github.com/apache/datafusion/pull/15181#issuecomment-2785491624
Looks like the security audit failure is due to #15571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc closed pull request #15181: Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider`
URL: https://github.com/apache/datafusion/pull/15181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org