Get stream of rejected data from Elasticsearch6 sink

[Nick Triller <Ni...@axoom.com>]

Hi all,

is it possible to further process data that could not be persisted by the Elasticsearch6 sink without breaking checkpointing?
As I understand, the onFailure callback can't be used to forward rejected data into a separate stream.

I would like to extend the sink if this use case is not covered yet.
What would be a reasonable approach that matches Flink's overall architecture?

As a new Flink user, my first intuition was to check if it is possible to create a side output from the Elasticsearch6 sink.
Does it make sense to support sink side outputs?

Thank you for your advice.

Regards,
Nick

Re: Get stream of rejected data from Elasticsearch6 sink

[vino yang <ya...@gmail.com>]

Hi Nick,

When you get the failed data, the logic for implementing the side output is
similar to the logic for extending the
ActionRequestFailureHandler#onFailure method to output the data to other
places.

Thanks, vino.

Nick Triller <Ni...@axoom.com> 于2018年8月31日周五 下午9:08写道：

> Hi all,
>
>
>
> is it possible to further process data that could not be persisted by the
> Elasticsearch6 sink without breaking checkpointing?
>
> As I understand, the onFailure callback can’t be used to forward rejected
> data into a separate stream.
>
>
>
> I would like to extend the sink if this use case is not covered yet.
>
> What would be a reasonable approach that matches Flink’s overall
> architecture?
>
>
>
> As a new Flink user, my first intuition was to check if it is possible to
> create a side output from the Elasticsearch6 sink.
>
> Does it make sense to support sink side outputs?
>
>
>
> Thank you for your advice.
>
>
>
> Regards,
>
> Nick
>