You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Victor Villa Dev <vi...@gmail.com> on 2019/11/24 18:27:18 UTC
Side output from Flink Sink
I'd like know if there's a way to generate a side output and/or sink to an
alternate kafka topic from within a Sink?
The use case is the datastream sinks to a storage and on particular failed
attempts I'd like to deadletter to a kafka topic.
Any suggestions?
Thanks
Re: Side output from Flink Sink
Posted by Robert Metzger <rm...@apache.org>.
What do you mean by "from within a sink"? Do you have a custom sink?
If you want to write to different Kafka topics from the same sink, you can
do that using a custom KafkaSerializationSchema. It allows you to return a
ProducerRecord with a custom target topic set. (A Kafka sink can write to
multiple topics).
Why can't you split the stream into "good" and "bad" records before the
sink, and then define two different sinks?
On Thu, Nov 28, 2019 at 1:41 PM Arvid Heise <ar...@ververica.com> wrote:
> Hi Victor,
>
> you could implement your own SinkFunction that wraps the KafkaProducer.
> However, since you may need to check if the write operation is successful,
> you probably need to subclass KafkaProducer and implement your own error
> handling.
>
> Best,
>
> Arvid
>
> On Mon, Nov 25, 2019 at 7:51 AM vino yang <ya...@gmail.com> wrote:
>
>> Hi Victor,
>>
>> Currently, it seems the "side output" feature does not been supported by
>> the streaming sink.
>>
>> IMO, you can customize your sink via selecting different types of events
>> to output to different places.
>>
>> WDYT?
>>
>> Best,
>> Vino
>>
>> Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 下午1:37写道：
>>
>>> Hi Vino,
>>>
>>> Thanks a lot for your reply!
>>> However I'm not quite sure my question was clear enough.
>>> I'm aware I can create/get side outputs using output tags from within
>>> operators (Process Functions) as documentation also states.
>>>
>>> The main point in my question is wether creating a sideo output is even
>>> possible from within a Sink?
>>> if so, would you mind pointing to an examples on how to correctly get
>>> the context necessary to add the "output" from within the "invoke()" method.
>>> In case it isn't what are the usual/suggested strategies?
>>>
>>> I know the Sink is usually the "last" portion of a data stream as its
>>> name indicates, but I was wondering if for some reason something can't be
>>> sinked (after retries, etc), what is the usual way to deal with such cases?
>>>
>>> Thanks again for your kind support.
>>>
>>> On 2019/11/25 02:23:15, vino yang <ya...@gmail.com> wrote:
>>> > Hi Victor,
>>> >
>>> > Firstly, you can get your side output stream via OutputTag. Please
>>> refer to
>>> > the official documentation[1].
>>> > Then, specify a sink for your side output stream. Of course, you can
>>> > specify a Kafka sink.
>>> >
>>> > Best,
>>> > Vino
>>> >
>>> > [1]:
>>> >
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>>> >
>>> > Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 上午2:27写道：
>>> >
>>> > > I'd like know if there's a way to generate a side output and/or sink
>>> to an
>>> > > alternate kafka topic from within a Sink?
>>> > > The use case is the datastream sinks to a storage and on particular
>>> failed
>>> > > attempts I'd like to deadletter to a kafka topic.
>>> > > Any suggestions?
>>> > >
>>> > > Thanks
>>> > >
>>> >
>>>
>>
Re: Side output from Flink Sink
Posted by Arvid Heise <ar...@ververica.com>.
Hi Victor,
you could implement your own SinkFunction that wraps the KafkaProducer.
However, since you may need to check if the write operation is successful,
you probably need to subclass KafkaProducer and implement your own error
handling.
Best,
Arvid
On Mon, Nov 25, 2019 at 7:51 AM vino yang <ya...@gmail.com> wrote:
> Hi Victor,
>
> Currently, it seems the "side output" feature does not been supported by
> the streaming sink.
>
> IMO, you can customize your sink via selecting different types of events
> to output to different places.
>
> WDYT?
>
> Best,
> Vino
>
> Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 下午1:37写道：
>
>> Hi Vino,
>>
>> Thanks a lot for your reply!
>> However I'm not quite sure my question was clear enough.
>> I'm aware I can create/get side outputs using output tags from within
>> operators (Process Functions) as documentation also states.
>>
>> The main point in my question is wether creating a sideo output is even
>> possible from within a Sink?
>> if so, would you mind pointing to an examples on how to correctly get the
>> context necessary to add the "output" from within the "invoke()" method.
>> In case it isn't what are the usual/suggested strategies?
>>
>> I know the Sink is usually the "last" portion of a data stream as its
>> name indicates, but I was wondering if for some reason something can't be
>> sinked (after retries, etc), what is the usual way to deal with such cases?
>>
>> Thanks again for your kind support.
>>
>> On 2019/11/25 02:23:15, vino yang <ya...@gmail.com> wrote:
>> > Hi Victor,
>> >
>> > Firstly, you can get your side output stream via OutputTag. Please
>> refer to
>> > the official documentation[1].
>> > Then, specify a sink for your side output stream. Of course, you can
>> > specify a Kafka sink.
>> >
>> > Best,
>> > Vino
>> >
>> > [1]:
>> >
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>> >
>> > Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 上午2:27写道：
>> >
>> > > I'd like know if there's a way to generate a side output and/or sink
>> to an
>> > > alternate kafka topic from within a Sink?
>> > > The use case is the datastream sinks to a storage and on particular
>> failed
>> > > attempts I'd like to deadletter to a kafka topic.
>> > > Any suggestions?
>> > >
>> > > Thanks
>> > >
>> >
>>
>
Re: Side output from Flink Sink
Posted by vino yang <ya...@gmail.com>.
Hi Victor,
Currently, it seems the "side output" feature does not been supported by
the streaming sink.
IMO, you can customize your sink via selecting different types of events to
output to different places.
WDYT?
Best,
Vino
Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 下午1:37写道：
> Hi Vino,
>
> Thanks a lot for your reply!
> However I'm not quite sure my question was clear enough.
> I'm aware I can create/get side outputs using output tags from within
> operators (Process Functions) as documentation also states.
>
> The main point in my question is wether creating a sideo output is even
> possible from within a Sink?
> if so, would you mind pointing to an examples on how to correctly get the
> context necessary to add the "output" from within the "invoke()" method.
> In case it isn't what are the usual/suggested strategies?
>
> I know the Sink is usually the "last" portion of a data stream as its name
> indicates, but I was wondering if for some reason something can't be sinked
> (after retries, etc), what is the usual way to deal with such cases?
>
> Thanks again for your kind support.
>
> On 2019/11/25 02:23:15, vino yang <ya...@gmail.com> wrote:
> > Hi Victor,
> >
> > Firstly, you can get your side output stream via OutputTag. Please refer
> to
> > the official documentation[1].
> > Then, specify a sink for your side output stream. Of course, you can
> > specify a Kafka sink.
> >
> > Best,
> > Vino
> >
> > [1]:
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
> >
> > Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 上午2:27写道：
> >
> > > I'd like know if there's a way to generate a side output and/or sink
> to an
> > > alternate kafka topic from within a Sink?
> > > The use case is the datastream sinks to a storage and on particular
> failed
> > > attempts I'd like to deadletter to a kafka topic.
> > > Any suggestions?
> > >
> > > Thanks
> > >
> >
>
Re: Side output from Flink Sink
Posted by Victor Villa Dev <vi...@gmail.com>.
Hi Vino,
Thanks a lot for your reply!
However I'm not quite sure my question was clear enough.
I'm aware I can create/get side outputs using output tags from within operators (Process Functions) as documentation also states.
The main point in my question is wether creating a sideo output is even possible from within a Sink?
if so, would you mind pointing to an examples on how to correctly get the context necessary to add the "output" from within the "invoke()" method.
In case it isn't what are the usual/suggested strategies?
I know the Sink is usually the "last" portion of a data stream as its name indicates, but I was wondering if for some reason something can't be sinked (after retries, etc), what is the usual way to deal with such cases?
Thanks again for your kind support.
On 2019/11/25 02:23:15, vino yang <ya...@gmail.com> wrote:
> Hi Victor,
>
> Firstly, you can get your side output stream via OutputTag. Please refer to
> the official documentation[1].
> Then, specify a sink for your side output stream. Of course, you can
> specify a Kafka sink.
>
> Best,
> Vino
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>
> Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 上午2:27写道：
>
> > I'd like know if there's a way to generate a side output and/or sink to an
> > alternate kafka topic from within a Sink?
> > The use case is the datastream sinks to a storage and on particular failed
> > attempts I'd like to deadletter to a kafka topic.
> > Any suggestions?
> >
> > Thanks
> >
>
Re: Side output from Flink Sink
Posted by vino yang <ya...@gmail.com>.
Hi Victor,
Firstly, you can get your side output stream via OutputTag. Please refer to
the official documentation[1].
Then, specify a sink for your side output stream. Of course, you can
specify a Kafka sink.
Best,
Vino
[1]:
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
Victor Villa Dev <vi...@gmail.com> 于2019年11月25日周一 上午2:27写道：
> I'd like know if there's a way to generate a side output and/or sink to an
> alternate kafka topic from within a Sink?
> The use case is the datastream sinks to a storage and on particular failed
> attempts I'd like to deadletter to a kafka topic.
> Any suggestions?
>
> Thanks
>