[jira] [Updated] (FLINK-17590) Add Bucket lifecycle listener to support acquiring bucket state

["Yun Gao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-17590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yun Gao updated FLINK-17590:
----------------------------
        Parent: FLINK-17589
    Issue Type: Sub-task  (was: New Feature)

> Add Bucket lifecycle listener to support acquiring bucket state
> ---------------------------------------------------------------
>
>                 Key: FLINK-17590
>                 URL: https://issues.apache.org/jira/browse/FLINK-17590
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Connectors / FileSystem
>            Reporter: Yun Gao
>            Priority: Major
>             Fix For: 1.11.0
>
>
> Hive sink will reuse the Buckets class of StreamingFileSink, which encapsulate most of the logic of StreamingFileSink. Hive sink requires to writing one-piece of meta-info into Hive meta store after a partition (namely Bucket in StreamingFileSink) has been terminated. Currently the termination is judged by event-time/processing time ([FLIP-115|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table]]).
>  
> To support the requirement of the Hive Sink, we would add listener for acquiring the event bucket creation and getting inactive. A bucket get inactive if all the previous records have been committed. Then Hive Sink could safely writing meta-info if the time has exceeded the bucket's boundary and it has been inactive. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)