[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #11844: ARROW-14972: [Python][Doc] Document automatic partitioning discovery

[GitBox <gi...@apache.org>]

jorisvandenbossche commented on a change in pull request #11844:
URL: https://github.com/apache/arrow/pull/11844#discussion_r783280265



##########
File path: docs/source/python/dataset.rst
##########
@@ -340,6 +340,30 @@ when constructing a directory partitioning:
 Directory partitioning also supports providing a full schema rather than inferring
 types from file paths.
 
+Automatic partitioning detection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the directory is partitioned using the hive partitioning scheme (see above)
+then pyarrow will be able to automatically recognize the partitioning and include
+the partitioning information as a column in the returned table.  There is no
+need to specify the partitioning unless you need to override the inferred data
+types of the partitioning columns:
+
+.. code-block:: python
+
+    dataset = ds.dataset("hive_partitioned", format="parquet")

Review comment:
       @AlenkaF and I ran into some similar issues again, reminding me that we should have better defaults (see also https://issues.apache.org/jira/browse/ARROW-15310 that I opened)
   
   Agreed that for the read side, we can certainly already switch the default, as that is quite harmless. 
   
   It would still be nice to have the write side to be consistent, but that's a bigger change. In theory we could do it with a deprecation cycle, though, now that we have the `partitioning_flavor` keyword. Currently that defaults to None, and you can set it to "hive" to get hive-style instead of directory partitioning. But so we could raise a warning if it is None (and a partitioning is specified), saying it will change to "hive" in the future, and people can specify "directory" or "hive" explicitly to silence the warning.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org