Re: Pending parquet file with Bucking Sink

[Aljoscha Krettek <al...@apache.org>]

Hi,

Your analysis is correct. If the program ends before we can do a checkpoint files will never be moved to "final" state. We could move all files to "final" stage when the Sink is closing but the problem here is that Flink currently doesn't provide a way for user functions (which Sinks are) to distinguish between "erroneous close" and "close because of stream end" so we cannot currently do this. We are aware of the problem and this is the Jira Issue for tracking it: https://issues.apache.org/jira/browse/FLINK-2646 <https://issues.apache.org/jira/browse/FLINK-2646>

Best,
Aljoscha 

> On 20. Dec 2017, at 19:05, xiatao123 <ta...@udacity.com> wrote:
> 
> Hi Vipul,
>  Thanks for the information.  Yes, I do have checkpointing enabled with 10
> millisecs.
>  I think the issue here is that the stream ended before the checkpoint
> reached.  This is a testing code that the DataStream only have 5 events then
> it ended. Once the stream ended, the checkpoint is not triggered, then the
> file remains in "pending" state.
>  Anyway we can force a checkpoint trigger? or let the sink know the stream
> ended? 
> Thanks,
> Tao
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/