You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Great Info <gu...@gmail.com> on 2023/01/05 15:48:10 UTC
flink add multiple sink in sequence
I have a stream from Kafka, after reading it and doing some
transformations/enrichment I need to store the final data stream in the
database and publish it to Kafka so I am planning to add two sinks like
below
*finalInputStream.addSink(dataBaseSink); // Sink1finalInputStream.addSink(
flinkKafkaProducer ); //Sink2 *
Has the sequence guaranteed between Sink1 and Sink2, in my requirement
stream to sink2 should start to begin only after successfully completing
Sink1, If Sink1 fails it should not write to Sink2.
Re: flink add multiple sink in sequence
Posted by Martijn Visser <ma...@apache.org>.
Correct. If you want to have guarantees between Sink1 and Sink2, you need
to create a combined sink
Op zo 8 jan. 2023 om 15:57 schreef Great Info <gu...@gmail.com>
> Does it mean sequence of writing is not guaranteed between these
> independent Sink1 and Sink2?
>
>
>
>
> On Fri, 6 Jan 2023, 10:44 am Shammon FY, <zj...@gmail.com> wrote:
>
>> Hi @Great
>>
>> I think the two sinks in your example are equivalent and independent. If
>> there are some logical relationships between two sinks, you may need to
>> create a new combined sink and do it yourself.
>>
>> On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>>
>>>
>>> I have a stream from Kafka, after reading it and doing some
>>> transformations/enrichment I need to store the final data stream in the
>>> database and publish it to Kafka so I am planning to add two sinks like
>>> below
>>>
>>>
>>> *finalInputStream.addSink(dataBaseSink); //
>>> Sink1finalInputStream.addSink( flinkKafkaProducer ); //Sink2 *
>>>
>>> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
>>> stream to sink2 should start to begin only after successfully completing
>>> Sink1, If Sink1 fails it should not write to Sink2.
>>>
>> --
Martijn
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
Re: flink add multiple sink in sequence
Posted by Martijn Visser <ma...@apache.org>.
Correct. If you want to have guarantees between Sink1 and Sink2, you need
to create a combined sink
Op zo 8 jan. 2023 om 15:57 schreef Great Info <gu...@gmail.com>
> Does it mean sequence of writing is not guaranteed between these
> independent Sink1 and Sink2?
>
>
>
>
> On Fri, 6 Jan 2023, 10:44 am Shammon FY, <zj...@gmail.com> wrote:
>
>> Hi @Great
>>
>> I think the two sinks in your example are equivalent and independent. If
>> there are some logical relationships between two sinks, you may need to
>> create a new combined sink and do it yourself.
>>
>> On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>>
>>>
>>> I have a stream from Kafka, after reading it and doing some
>>> transformations/enrichment I need to store the final data stream in the
>>> database and publish it to Kafka so I am planning to add two sinks like
>>> below
>>>
>>>
>>> *finalInputStream.addSink(dataBaseSink); //
>>> Sink1finalInputStream.addSink( flinkKafkaProducer ); //Sink2 *
>>>
>>> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
>>> stream to sink2 should start to begin only after successfully completing
>>> Sink1, If Sink1 fails it should not write to Sink2.
>>>
>> --
Martijn
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
Re: flink add multiple sink in sequence
Posted by Great Info <gu...@gmail.com>.
Does it mean sequence of writing is not guaranteed between these
independent Sink1 and Sink2?
On Fri, 6 Jan 2023, 10:44 am Shammon FY, <zj...@gmail.com> wrote:
> Hi @Great
>
> I think the two sinks in your example are equivalent and independent. If
> there are some logical relationships between two sinks, you may need to
> create a new combined sink and do it yourself.
>
> On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>
>>
>> I have a stream from Kafka, after reading it and doing some
>> transformations/enrichment I need to store the final data stream in the
>> database and publish it to Kafka so I am planning to add two sinks like
>> below
>>
>>
>> *finalInputStream.addSink(dataBaseSink); //
>> Sink1finalInputStream.addSink( flinkKafkaProducer ); //Sink2 *
>>
>> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
>> stream to sink2 should start to begin only after successfully completing
>> Sink1, If Sink1 fails it should not write to Sink2.
>>
>
Re: flink add multiple sink in sequence
Posted by Great Info <gu...@gmail.com>.
Does it mean sequence of writing is not guaranteed between these
independent Sink1 and Sink2?
On Fri, 6 Jan 2023, 10:44 am Shammon FY, <zj...@gmail.com> wrote:
> Hi @Great
>
> I think the two sinks in your example are equivalent and independent. If
> there are some logical relationships between two sinks, you may need to
> create a new combined sink and do it yourself.
>
> On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>
>>
>> I have a stream from Kafka, after reading it and doing some
>> transformations/enrichment I need to store the final data stream in the
>> database and publish it to Kafka so I am planning to add two sinks like
>> below
>>
>>
>> *finalInputStream.addSink(dataBaseSink); //
>> Sink1finalInputStream.addSink( flinkKafkaProducer ); //Sink2 *
>>
>> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
>> stream to sink2 should start to begin only after successfully completing
>> Sink1, If Sink1 fails it should not write to Sink2.
>>
>
Re: flink add multiple sink in sequence
Posted by Shammon FY <zj...@gmail.com>.
Hi @Great
I think the two sinks in your example are equivalent and independent. If
there are some logical relationships between two sinks, you may need to
create a new combined sink and do it yourself.
On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>
> I have a stream from Kafka, after reading it and doing some
> transformations/enrichment I need to store the final data stream in the
> database and publish it to Kafka so I am planning to add two sinks like
> below
>
>
> *finalInputStream.addSink(dataBaseSink); // Sink1finalInputStream.addSink(
> flinkKafkaProducer ); //Sink2 *
>
> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
> stream to sink2 should start to begin only after successfully completing
> Sink1, If Sink1 fails it should not write to Sink2.
>
Re: flink add multiple sink in sequence
Posted by Shammon FY <zj...@gmail.com>.
Hi @Great
I think the two sinks in your example are equivalent and independent. If
there are some logical relationships between two sinks, you may need to
create a new combined sink and do it yourself.
On Thu, Jan 5, 2023 at 11:48 PM Great Info <gu...@gmail.com> wrote:
>
> I have a stream from Kafka, after reading it and doing some
> transformations/enrichment I need to store the final data stream in the
> database and publish it to Kafka so I am planning to add two sinks like
> below
>
>
> *finalInputStream.addSink(dataBaseSink); // Sink1finalInputStream.addSink(
> flinkKafkaProducer ); //Sink2 *
>
> Has the sequence guaranteed between Sink1 and Sink2, in my requirement
> stream to sink2 should start to begin only after successfully completing
> Sink1, If Sink1 fails it should not write to Sink2.
>