You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/04/21 07:16:48 UTC
[PR] Skip files outside partition structure in hive-partitioned listing tables [datafusion]
zhuqi-lucas opened a new pull request, #21756:
URL: https://github.com/apache/datafusion/pull/21756
## Which issue does this PR close?
Closes #21755
## Rationale
When a hive-partitioned listing table has files in the root directory (not inside any `partition_col=value/` path), queries that reference partition columns crash with `Unable to get field named "partition_col"`.
This is a common scenario when a table transitions from non-partitioned to hive-partitioned storage — the original root file may still exist alongside the new partition directories.
## What changes are included in this PR?
`try_into_partitioned_file` now returns `Ok(None)` for files that don't match the partition structure (where `parse_partitions_for_path` returns `None`). The caller skips them via `try_filter_map`.
Previously, `None` from `parse_partitions_for_path` was converted to empty `partition_values` via `.into_iter().flatten()`, causing downstream errors.
## Are these changes tested?
Yes, 5 unit tests added:
- `test_try_into_partitioned_file_valid_partition` — normal case
- `test_try_into_partitioned_file_root_file_skipped` — root file skipped
- `test_try_into_partitioned_file_wrong_partition_name` — wrong partition col name
- `test_try_into_partitioned_file_multiple_partitions` — multi-level partitions
- `test_try_into_partitioned_file_partial_partition_skipped` — incomplete partition path
## Are there any user-facing changes?
Files outside the hive partition structure are now silently skipped instead of causing query failures. This may change `COUNT(*)` results if such files previously contributed rows (with empty partition values).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Skip files outside partition structure in hive-partitioned listing tables [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #21756:
URL: https://github.com/apache/datafusion/pull/21756#discussion_r3115713255
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -340,17 +340,28 @@ fn filter_partitions(
Ok(None)
}
+/// Returns `Ok(None)` when the file is not inside a valid partition path
+/// (e.g. a stale file in the table root directory). Such files are skipped
+/// because hive-style partition values are never null and there is no valid
+/// value to assign for non-partitioned files.
fn try_into_partitioned_file(
object_meta: ObjectMeta,
partition_cols: &[(String, DataType)],
table_path: &ListingTableUrl,
-) -> Result<PartitionedFile> {
+) -> Result<Option<PartitionedFile>> {
let cols = partition_cols.iter().map(|(name, _)| name.as_str());
let parsed = parse_partitions_for_path(table_path, &object_meta.location, cols);
+ let Some(parsed) = parsed else {
+ debug!(
+ "Skipping file outside partition structure: {}",
+ object_meta.location
+ );
Review Comment:
This `debug!` log will often duplicate the `debug!` emitted inside `parse_partitions_for_path` for the same file (it already logs on partition mismatch). To avoid double logging/noisy listings, consider either removing this log, lowering it to `trace!`, or logging only for the `strip_prefix`-failure case (which currently returns `None` without logging).
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Skip files outside partition structure in hive-partitioned listing tables [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas merged PR #21756:
URL: https://github.com/apache/datafusion/pull/21756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Skip files outside partition structure in hive-partitioned listing tables [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #21756:
URL: https://github.com/apache/datafusion/pull/21756#issuecomment-4293321167
Thanks @xudong963 and @alamb for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Skip files outside partition structure in hive-partitioned listing tables [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #21756:
URL: https://github.com/apache/datafusion/pull/21756#discussion_r3121004956
##########
datafusion/sqllogictest/test_files/listing_table_partitions.slt:
##########
@@ -73,3 +73,70 @@ foo
statement count 0
set datafusion.execution.listing_table_factory_infer_partitions = true;
+
+# Test: files outside partition structure are skipped
Review Comment:
💯 for lt tests 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org