[jira] [Commented] (FLINK-17900) Listener for when Files are committed in StreamingFileSink

["Guowei Ma (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17115629#comment-17115629 ] 

Guowei Ma commented on FLINK-17900:
-----------------------------------

Hi, [~maccamlc]
 # Currently, there is no public API that could let you register your own listener. 
 # Do you need to be notified every time when a file is committed? or to only be notified when a bucket is updated?

thanks.

> Listener for when Files are committed in StreamingFileSink
> ----------------------------------------------------------
>
>                 Key: FLINK-17900
>                 URL: https://issues.apache.org/jira/browse/FLINK-17900
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: 1.10.0
>            Reporter: Matthew McMahon
>            Priority: Major
>
> I have a scenario where once a file has been committed to S3 using the StreamingFileSink (I am using Java), that I should notify some downstream services.
> The idea is to produce a message on a Kafka topic for the files as they are finalized.
> I am currently looking into how this might be possible, and considering using reflection and/or checking the S3 bucket before/after the checkpoint is committed.
> Still trying to find a suitable way.
> However I was thinking it would be great if possible to register a listener that can be fired when StreamingFileSink commits a file. 
> Does something like this exist, that I'm not aware of (new to flink) or could it be added?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)