You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "BlakeOrth (via GitHub)" <gi...@apache.org> on 2025/08/05 22:47:53 UTC
[I] `ListingTableFactory` yields unexpected results for Hive partitioned datasets [datafusion]
BlakeOrth opened a new issue, #17049:
URL: https://github.com/apache/datafusion/issues/17049
### Describe the bug
When using a `ListingTableFactory` (i.e. using the `datafusion-cli`) tables that leverage Hive partitioning do not yield the expected columns derived from the hive key/value pairs (and other such oddities).
### To Reproduce
A publicly accessible S3 bucket hosting Hive partitioned parquet data can be used to show the issues. The general structure of the data can be seen below:
```console
$ aws s3 ls s3://overturemaps-us-west-2/release/2025-07-23.0/
PRE theme=addresses/
PRE theme=base/
PRE theme=buildings/
PRE theme=divisions/
PRE theme=places/
PRE theme=transportation/
```
```console
$ aws s3 ls s3://overturemaps-us-west-2/release/2025-07-23.0/ --recursive
2025-07-22 14:42:50 1073537247 release/2025-07-23.0/theme=addresses/type=address/part-00000-57f746d8-98a1-4faa-94c2-4f084343ecac-c000.zstd.parquet
2025-07-22 14:43:06 1013189887 release/2025-07-23.0/theme=addresses/type=address/part-00001-57f746d8-98a1-4faa-94c2-4f084343ecac-c000.zstd.parquet
. . .
```
Here are a couple of examples off odd behavior when trying to access this data.
```sql
DataFusion CLI v49.0.0
> SET datafusion.execution.listing_table_ignore_subdirectory to false;
0 row(s) fetched.
Elapsed 0.001 seconds.
> CREATE EXTERNAL TABLE overture_maps
STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-07-23.0/';
0 row(s) fetched.
Elapsed 0.645 seconds.
> describe overture_maps;
+-------------+-----------+-------------+
| column_name | data_type | is_nullable |
+-------------+-----------+-------------+
+-------------+-----------+-------------+
0 row(s) fetched.
Elapsed 0.001 seconds.
```
In the above example no schema has been detected at all, either from Hive keys or the underlying parquet data itself, in spite of ensuring the listing table can traverse subdirectories.
```sql
> CREATE EXTERNAL TABLE overture_maps
STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-07-23.0/theme=addresses/';
0 row(s) fetched.
Elapsed 0.911 seconds.
> select column_name, is_nullable from information_schema.columns where table_name='overture_maps';
+----------------+-------------+
| column_name | is_nullable |
+----------------+-------------+
| id | YES |
| geometry | YES |
| bbox | YES |
| country | YES |
| postcode | YES |
| street | YES |
| number | YES |
| unit | YES |
| address_levels | YES |
| postal_city | YES |
| version | YES |
| sources | YES |
+----------------+-------------+
12 row(s) fetched.
Elapsed 0.005 seconds.
> select column_name, is_nullable from information_schema.columns where table_name='overture_maps' and column_name='type';
+-------------+-------------+
| column_name | is_nullable |
+-------------+-------------+
+-------------+-------------+
0 row(s) fetched.
Elapsed 0.006 seconds.
```
Next, when a path to a subdirectory is given to create the table, the underlying parquet schema is successfully detected, however the columns corresponding to the Hive keys are still missing.
### Expected behavior
When accessing a hive partitioned dataset with no explicit schema set, and having subdirectory traversal enabled, I would expect to get a table that can be queried/filtered on both the columns represented in the parquet schema as well as the hive partitions.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Auto detecting partitions with `ListingTableFactory` on Hive partitioned datasets [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17049: Auto detecting partitions with `ListingTableFactory` on Hive partitioned datasets
URL: https://github.com/apache/datafusion/issues/17049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Bug auto detecting partitions with `ListingTableFactory` on Hive partitioned datasets [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #17049:
URL: https://github.com/apache/datafusion/issues/17049#issuecomment-3192236107
@alamb I think the updated title is more descriptive. It does lose the notion that, even in non-partitioned datasets, paths with a `.` in the final element could fail to discover any data at all. I could also see that being filed as a separate bug, at the time I opened this I actually didn't even realize that was part of the problems I saw here and only discovered it as I was implementing the fix for discovery. Should I open a separate bug for that and associate apache/datafusion/pull/17050 with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Bug auto detecting partitions with `ListingTableFactory` on Hive partitioned datasets [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17049:
URL: https://github.com/apache/datafusion/issues/17049#issuecomment-3191702595
I updated the description of this ticket to better reflect what I think the problem is - let me know if that doesn't make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Bug auto detecting partitions with `ListingTableFactory` on Hive partitioned datasets [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17049:
URL: https://github.com/apache/datafusion/issues/17049#issuecomment-3192732752
> [@alamb](https://github.com/alamb) I think the updated title is more descriptive. It does lose the notion that, even in non-partitioned datasets, paths with a `.` in the final element could fail to discover any data at all. I could also see that being filed as a separate bug, at the time I opened this I actually didn't even realize that was part of the problems I saw here and only discovered it as I was implementing the fix for discovery. Should I open a separate bug for that and associate [/pull/17050](https://github.com/apache/datafusion/pull/17050) with it?
yes please -- if you think it is a separate symptom with a separate fix, that would be a good new ticket to file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org