[jira] [Commented] (ARROW-14149) Support a "modified" hive style directory naming scheme

["Ryan Hafen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-14149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421104#comment-17421104 ] 

Ryan Hafen commented on ARROW-14149:
------------------------------------

Would one option here be that when you don't use hive-style names, then the partitioning variables automatically get stored as metadata when you write the file and then when you read it, it looks for that metadata and specifies the partitioning variables for you? I'm thinking of something like this being implemented in the high-level functions like write_dataset() and open_dataset() in R for example.

> Support a "modified" hive style directory naming scheme
> -------------------------------------------------------
>
>                 Key: ARROW-14149
>                 URL: https://issues.apache.org/jira/browse/ARROW-14149
>             Project: Apache Arrow
>          Issue Type: Improvement
>            Reporter: Ryan Hafen
>            Priority: Minor
>
> I am working on a project where I need to create and analyze parquet files using Apache Arrow but the environment I'm working with does not allow "=" in file paths, which the hive naming convention forces, e.g. "year=2007". While I can specify the partitioning to not use the hive contention, I then lose the variable names. This is problematic when I'm sharing the datasets with others because they will have to specify the partitioning variables when opening the dataset but they don't know what the partitioning variables are.
>  
> Would it be possible to allow a modified hive-style directory naming convention that still preserves the variable name in the directory name? For example, allowing a delimiter other than "="?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)