You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "Chao Long (Jira)" <ji...@apache.org> on 2019/09/03 03:03:00 UTC

[jira] [Resolved] (KYLIN-3626) Allow customization for Parquet path

     [ https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chao Long resolved KYLIN-3626.
------------------------------
    Resolution: Resolved

> Allow customization for Parquet path
> ------------------------------------
>
>                 Key: KYLIN-3626
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3626
>             Project: Kylin
>          Issue Type: Sub-task
>          Components: Job Engine, Metadata, Storage - Parquet
>            Reporter: Shaofeng SHI
>            Assignee: Chao Long
>            Priority: Major
>
> Due to the difference of HDFS and object store like S3/blob store, Kylin needs to allow different path patterns for the files when putting to the underlying storage. 
>  
> For example, to HDFS, this pattern is good:
> /kylin/metadata_name/cube_name/segment_name/file1.parquet
>  
> But, the above pattern is not good for S3, since S3 uses the first couple bytes of the path to do the hashing, so we'd better add salt ahead of the original pathname:
> /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet
> or do a reverse order:
> /segment_name/cube_name/metadata_name/kylin/file1.parquet
>  
> To get the flexibility, we should make this extensible.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)