You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Mohamed Amine ABDESSEMED (JIRA)" <ji...@apache.org> on 2015/09/15 10:42:46 UTC
[jira] [Created] (FLINK-2672) Add partitioned output format to HDFS
RollingSink
Mohamed Amine ABDESSEMED created FLINK-2672:
-----------------------------------------------
Summary: Add partitioned output format to HDFS RollingSink
Key: FLINK-2672
URL: https://issues.apache.org/jira/browse/FLINK-2672
Project: Flink
Issue Type: Improvement
Components: Streaming Connectors
Affects Versions: 0.10
Reporter: Mohamed Amine ABDESSEMED
Priority: Minor
An interesting use case of the HDFS Sink is to dispatch data into multiple directories depending of attributes present in the source data.
For example, for some data with a timestamp and a status fields, we want to write it into different directories using a pattern like : /somepath/%{timestamp}/%{status}
The expected results are somethings like:
/somepath/some_timestamp/wellformed
/somepath/some_timestamp/malformed
/somepath/some_timestamp/incomplete
...
etc
To support this functionality the bucketing and checkpointing logics need to be changed.
Note: For now, this can be done using the current version of the Rolling HDFS Sink with the help of splitting data streams and having multiple HDFS sinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)