[jira] [Comment Edited] (FLINK-9411) Support parquet rolling sink writer

["Triones Deng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519405#comment-16519405 ] 

Triones Deng edited comment on FLINK-9411 at 6/21/18 2:29 PM:
--------------------------------------------------------------

[~StephanEwen] sure a design is necessary, here there may be two ways to do it i think
in production. Orc and Parquet file is popupar because of columnar storage.
there are two method to support parquet writer.
1. just write a ParquetStreamWriter which is a subclass of StreamWriterBase. which looks the same as FLINK-9407
2. with a HdfsWriterWrapper which own one delegate writer, when end user want to use one format, just simply to specify the format like orc, parquet and let the wrapper create a suitable writer like OrcStreamWriter or ParquetStreamWriter and so on.
sample code for the HdfsWriterWrapper

{code:java}
public class HdfsWriterWrapper<T> implements Writer<T> {

    private Writer<T> delegate;
    private String format;
    private Configuration configuration;
    private TableSchema tableSchema;

    public HdfsWriterWrapper(Configuration configuration, String format,Class<T> tableClass,String[] columnFields){

    }
{code}
which one is better?


was (Author: triones):
[~StephanEwen] sure a design is necessary, here there may be two ways to do it i think
in production. Orc and Parquet file is popupar because of columnar storage.
there are two method to support parquet writer.
1. just write a ParquetStreamWriter which is a subclass of StreamWriterBase. which looks the same as FLINK-9407
2. with a HdfsWriterWrapper which own one delegate writer, when end user want to use one format, just simply to specify the format like orc, parquet and let the wrapper create a suitable writer like OrcStreamWriter or ParquetStreamWriter and so on.
sample code for the HdfsWriterWrapper

{code:java}
public class HdfsWriterWrapper<T> implements Writer<T> {

    private Writer<T> delegate;
    private String format;
    private Configuration configuration;
    private TableSchema tableSchema;

    public HdfsWriterWrapper(Configuration configuration, String format,Class<T> tableClass,String[] columnFields){

    }
{code}
what do you think will be better?

> Support parquet rolling sink writer
> -----------------------------------
>
>                 Key: FLINK-9411
>                 URL: https://issues.apache.org/jira/browse/FLINK-9411
>             Project: Flink
>          Issue Type: New Feature
>          Components: filesystem-connector
>            Reporter: mingleizhang
>            Assignee: Triones Deng
>            Priority: Major
>
> Like support orc rolling sink writer in FLINK-9407 , we should also support parquet rolling sink writer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)