[jira] [Updated] (KYLIN-3626) Allow customization for Parquet path

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shaofeng SHI updated KYLIN-3626:
--------------------------------
    Fix Version/s: v2.6.0

> Allow customization for Parquet path
> ------------------------------------
>
>                 Key: KYLIN-3626
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3626
>             Project: Kylin
>          Issue Type: Sub-task
>          Components: Metadata
>            Reporter: Shaofeng SHI
>            Assignee: Chao Long
>            Priority: Major
>             Fix For: v2.6.0
>
>
> Due to the difference of HDFS and object store like S3/blob store, Kylin needs to allow different path patterns for the files when putting to the underlying storage. 
>  
> For example, to HDFS, this pattern is good:
> /kylin/metadata_name/cube_name/segment_name/file1.parquet
>  
> But, the above pattern is not good for S3, since S3 uses the first couple bytes of the path to do the hashing, so we'd better add salt ahead of the original pathname:
> /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet
> or do a reverse order:
> /segment_name/cube_name/metadata_name/kylin/file1.parquet
>  
> To get the flexibility, we should make this extensible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)