You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Avi Levi <av...@bluevoyant.com> on 2019/12/01 09:02:23 UTC
Re: Idiomatic way to split pipeline
Thanks Arvid,
The problem is that I will get an exception on non unique uid on the
*stream* .
On Thu, Nov 28, 2019 at 2:45 PM Arvid Heise <ar...@ververica.com> wrote:
> *This Message originated outside your organization.*
> ------------------------------
> Hi Avi,
>
> it seems to me that you are not really needing any split feature. As far
> as I can see in your picture you want to apply two different windows on the
> same input data.
>
> In that case you simply use two different subgraphs.
>
> stream = ...
>
> stream1 = stream.window(...).....addSink(<sink1>)
>
> stream2 = stream.window(...).....addSink(<sink2>)
>
> In Flink, you can compose arbitrary directed acyclic graphs, so consuming
> the output of one operator on several downstream operators is completely
> normal.
>
> Best,
>
> Arvid
>
> On Mon, Nov 25, 2019 at 10:50 AM Avi Levi <av...@bluevoyant.com> wrote:
>
>> Thanks, I'll check it out.
>>
>> On Mon, Nov 25, 2019 at 11:46 AM vino yang <ya...@gmail.com> wrote:
>>
>>> *This Message originated outside your organization.*
>>> ------------------------------
>>> Hi Avi,
>>>
>>> The side output provides a superset of split's functionality. So
>>> anything can be implemented via split also can be implemented via side
>>> output.[1]
>>>
>>> Best,
>>> Vino
>>>
>>> [1]:
>>> https://stackoverflow.com/questions/51440677/apache-flink-whats-the-difference-between-side-outputs-and-split-in-the-data
>>> <https://stackoverflow.com/questions/51440677/apache-flink-whats-the-difference-between-side-outputs-and-split-in-the-data>
>>>
>>> Avi Levi <av...@bluevoyant.com> 于2019年11月25日周一 下午5:32写道:
>>>
>>>> Thank you, for your quick reply. I appreciate that. but this it not
>>>> exactly "side output" per se. it is simple splitting. IIUC The side output
>>>> is more for splitting the records buy something the differentiate them
>>>> (latnes , value etc' ) . I thought there is more idiomatic but if this is
>>>> it, than I will go with that.
>>>>
>>>> On Mon, Nov 25, 2019 at 10:42 AM vino yang <ya...@gmail.com>
>>>> wrote:
>>>>
>>>>> *This Message originated outside your organization.*
>>>>> ------------------------------
>>>>> Hi Avi,
>>>>>
>>>>> As the doc of DataStream#split said, you can use the "side output"
>>>>> feature to replace it.[1]
>>>>>
>>>>> [1]:
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>>>>> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html>
>>>>>
>>>>> Best,
>>>>> Vino
>>>>>
>>>>> Avi Levi <av...@bluevoyant.com> 于2019年11月25日周一 下午4:12写道:
>>>>>
>>>>>> Hi,
>>>>>> I want to split the output of one of the operators to two pipelines.
>>>>>> Since the *split* method is deprecated, what is the idiomatic way to
>>>>>> do that without duplicating the operator ?
>>>>>>
>>>>>> [image: Screen Shot 2019-11-25 at 10.05.38.png]
>>>>>>
>>>>>>
>>>>>>
Re: Idiomatic way to split pipeline
Posted by Robert Metzger <rm...@apache.org>.
Hi Avi,
can you post the exception with the stack trace here as well?
On Sun, Dec 1, 2019 at 10:03 AM Avi Levi <av...@bluevoyant.com> wrote:
> Thanks Arvid,
> The problem is that I will get an exception on non unique uid on the
> *stream* .
>
> On Thu, Nov 28, 2019 at 2:45 PM Arvid Heise <ar...@ververica.com> wrote:
>
>> *This Message originated outside your organization.*
>> ------------------------------
>> Hi Avi,
>>
>> it seems to me that you are not really needing any split feature. As far
>> as I can see in your picture you want to apply two different windows on the
>> same input data.
>>
>> In that case you simply use two different subgraphs.
>>
>> stream = ...
>>
>> stream1 = stream.window(...).....addSink(<sink1>)
>>
>> stream2 = stream.window(...).....addSink(<sink2>)
>>
>> In Flink, you can compose arbitrary directed acyclic graphs, so consuming
>> the output of one operator on several downstream operators is completely
>> normal.
>>
>> Best,
>>
>> Arvid
>>
>> On Mon, Nov 25, 2019 at 10:50 AM Avi Levi <av...@bluevoyant.com>
>> wrote:
>>
>>> Thanks, I'll check it out.
>>>
>>> On Mon, Nov 25, 2019 at 11:46 AM vino yang <ya...@gmail.com>
>>> wrote:
>>>
>>>> *This Message originated outside your organization.*
>>>> ------------------------------
>>>> Hi Avi,
>>>>
>>>> The side output provides a superset of split's functionality. So
>>>> anything can be implemented via split also can be implemented via side
>>>> output.[1]
>>>>
>>>> Best,
>>>> Vino
>>>>
>>>> [1]:
>>>> https://stackoverflow.com/questions/51440677/apache-flink-whats-the-difference-between-side-outputs-and-split-in-the-data
>>>>
>>>> Avi Levi <av...@bluevoyant.com> 于2019年11月25日周一 下午5:32写道:
>>>>
>>>>> Thank you, for your quick reply. I appreciate that. but this it not
>>>>> exactly "side output" per se. it is simple splitting. IIUC The side output
>>>>> is more for splitting the records buy something the differentiate them
>>>>> (latnes , value etc' ) . I thought there is more idiomatic but if this is
>>>>> it, than I will go with that.
>>>>>
>>>>> On Mon, Nov 25, 2019 at 10:42 AM vino yang <ya...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> *This Message originated outside your organization.*
>>>>>> ------------------------------
>>>>>> Hi Avi,
>>>>>>
>>>>>> As the doc of DataStream#split said, you can use the "side output"
>>>>>> feature to replace it.[1]
>>>>>>
>>>>>> [1]:
>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>>>>>>
>>>>>> Best,
>>>>>> Vino
>>>>>>
>>>>>> Avi Levi <av...@bluevoyant.com> 于2019年11月25日周一 下午4:12写道:
>>>>>>
>>>>>>> Hi,
>>>>>>> I want to split the output of one of the operators to two pipelines.
>>>>>>> Since the *split* method is deprecated, what is the idiomatic way
>>>>>>> to do that without duplicating the operator ?
>>>>>>>
>>>>>>> [image: Screen Shot 2019-11-25 at 10.05.38.png]
>>>>>>>
>>>>>>>
>>>>>>>