[jira] [Updated] (FLINK-2672) Add partitioned output format to HDFS RollingSink

["Robert Metzger (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Metzger updated FLINK-2672:
----------------------------------
    Component/s: filesystem-connector

> Add partitioned output format to HDFS RollingSink
> -------------------------------------------------
>
>                 Key: FLINK-2672
>                 URL: https://issues.apache.org/jira/browse/FLINK-2672
>             Project: Flink
>          Issue Type: Improvement
>          Components: filesystem-connector, Streaming Connectors
>    Affects Versions: 0.10.0
>            Reporter: Mohamed Amine ABDESSEMED
>            Priority: Minor
>              Labels: features
>
> An interesting use case of the HDFS Sink is to dispatch data into multiple directories depending of attributes present in the source data.
> For example, for some data with a timestamp and a status fields, we want to write it into different directories using a pattern like : /somepath/%{timestamp}/%{status}
> The expected results are somethings like: 
> /somepath/some_timestamp/wellformed
> /somepath/some_timestamp/malformed
> /somepath/some_timestamp/incomplete 
> ... 
> etc
> To support this functionality the bucketing and checkpointing logics need to be changed. 
> Note: For now, this can be done using the current version of the Rolling HDFS Sink (https://github.com/apache/flink/pull/1084) with the help of splitting data streams and having multiple HDFS sinks  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)