You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "greedAuguria (via GitHub)" <gi...@apache.org> on 2026/01/05 13:20:01 UTC
[PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
greedAuguria opened a new pull request, #19651:
URL: https://github.com/apache/datafusion/pull/19651
## Which issue does this PR close?
- Closes #19650.
## Rationale for this change
Currently, when DataFusion parses Hive-style partitioned paths (e.g., `s3://bucket/table/city=San%20Francisco/`), it extracts the partition value literally as `San%20Francisco`.
Standard practice in tools like Apache Spark and Apache Hive is to URL-encode special characters in partition values when writing to object stores. This PR ensures DataFusion correctly decodes these values (to `San Francisco`) during the listing process, preventing data mismatches and ensuring consistent behavior with other engines.
## What changes are included in this PR?
1. **Logic Update**: Updated `parse_partitions_for_path` in `datafusion/catalog-listing/src/helpers.rs` to percent-decode partition values.
2. **Signature Change**: Changed the return type of `parse_partitions_for_path` from `Option<Vec<&str>>` to `Option<Vec<String>>` because decoded strings require new allocations.
3. **Call Site Updates**: Updated internal callers in `datafusion-catalog-listing` to accommodate the owned `String` return type.
4. **Dependencies**: Added `percent-encoding` crate to `datafusion-catalog-listing`.
## Are these changes tested?
Yes, new unit tests were added to `datafusion/catalog-listing/src/helpers.rs` covering:
- Standard URL-encoded characters (e.g., `%2F` for `/`).
- Spaces encoded as `%20`.
- Multi-byte UTF-8 characters.
- Forgiving behavior for malformed encoding (matching `percent-encoding` crate defaults).
## Are there any user-facing changes?
Yes:
- **Data Behavior**: Partition values extracted from file paths will now be correctly decoded instead of remaining URL-encoded.
- **API Change**: The public helper function `parse_partitions_for_path` now returns `Vec<String>` instead of `Vec<&str>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19651:
URL: https://github.com/apache/datafusion/pull/19651#discussion_r2664717641
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -417,12 +418,15 @@ pub async fn pruned_partition_list<'a>(
}
/// Extract the partition values for the given `file_path` (in the given `table_path`)
-/// associated to the partitions defined by `table_partition_cols`
+/// associated to the partitions defined by `table_partition_cols`.
+///
+/// Partition values are URL-decoded, since object stores like S3 encode special
+/// characters (e.g., `/` becomes `%2F`) in path segments.
pub fn parse_partitions_for_path<'a, I>(
table_path: &ListingTableUrl,
file_path: &'a Path,
table_partition_cols: I,
-) -> Option<Vec<&'a str>>
+) -> Option<Vec<String>>
Review Comment:
nit: Not sure whether it is worth it to return a `Vec<Cow<'a, str>>` here ?! E.g. if the partition contains `%` then decode it and return `Cow::Owned(decoded.into_owned())`, otherwise `Cow::Borrowed(val).
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -417,12 +418,15 @@ pub async fn pruned_partition_list<'a>(
}
/// Extract the partition values for the given `file_path` (in the given `table_path`)
-/// associated to the partitions defined by `table_partition_cols`
+/// associated to the partitions defined by `table_partition_cols`.
+///
+/// Partition values are URL-decoded, since object stores like S3 encode special
+/// characters (e.g., `/` becomes `%2F`) in path segments.
pub fn parse_partitions_for_path<'a, I>(
table_path: &ListingTableUrl,
file_path: &'a Path,
table_partition_cols: I,
-) -> Option<Vec<&'a str>>
+) -> Option<Vec<String>>
Review Comment:
Actually this is what https://docs.rs/percent-encoding/latest/percent_encoding/struct.PercentDecode.html#method.decode_utf8 returns. It might be a good idea.
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -431,7 +435,10 @@ where
let mut part_values = vec![];
for (part, expected_partition) in subpath.zip(table_partition_cols) {
match part.split_once('=') {
- Some((name, val)) if name == expected_partition => part_values.push(val),
+ Some((name, val)) if name == expected_partition => {
+ let decoded = percent_decode_str(val).decode_utf8().ok()?;
Review Comment:
Should invalid values (like `%FF` in one of the tests below) return None or `val` ?!
I think it should behave like `%XX` - i.e. return `val`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
Posted by "greedAuguria (via GitHub)" <gi...@apache.org>.
greedAuguria commented on code in PR #19651:
URL: https://github.com/apache/datafusion/pull/19651#discussion_r3035324678
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -431,7 +435,10 @@ where
let mut part_values = vec![];
for (part, expected_partition) in subpath.zip(table_partition_cols) {
match part.split_once('=') {
- Some((name, val)) if name == expected_partition => part_values.push(val),
+ Some((name, val)) if name == expected_partition => {
+ let decoded = percent_decode_str(val).decode_utf8().ok()?;
Review Comment:
@martin-g I am sorry for the late reply, I will get on this now! Thank you for taking a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #19651: fix: Percent Encoding of paths for hive style partitioning
URL: https://github.com/apache/datafusion/pull/19651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
Posted by "greedAuguria (via GitHub)" <gi...@apache.org>.
greedAuguria commented on PR #19651:
URL: https://github.com/apache/datafusion/pull/19651#issuecomment-4186765500
@martin-g hey, sorry again for the late reply! I've rebased on main and addressed both of your comments:
**Cow<'a, str> instead of String** — you were right, `percent_decode_str().decode_utf8()` already hands back a `Cow<str>`, so it's a natural fit. Now we only allocate when the value actually had encoded characters in it. Most partition values won't, so this should be a nice win for the common case. Updated the caller to use `into_owned()` instead of `to_string()` too so we don't double-allocate on the owned path.
**Fallback for invalid encodings** — good catch on the `%FF` case. Previously `.ok()?` would bail out and return `None` for the whole file, which silently drops it from query results — definitely not what we want. Now if percent-decoding produces invalid UTF-8, we just pass through the raw value as-is (via `unwrap_or(Cow::Borrowed(val))`). So `%FF` comes back as the literal string `%FF` instead of the file disappearing. Same idea as your suggestion to treat it like `%XX`.
The taplo CI failure was just an alphabetical ordering thing in the root `Cargo.toml` — fixed during the rebase.
Let me know if there's anything else!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Percent Encoding of paths for hive style partitioning [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19651:
URL: https://github.com/apache/datafusion/pull/19651#issuecomment-4618380716
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org