You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Biplob Biswas <re...@gmail.com> on 2016/06/26 10:55:28 UTC
Way to hold execution of one of the map operator in Co-FlatMaps
Hi,
I was wondering whether it is possible to stop streaming data in from one of
the map operators until some data arrives in the second map operator.
For ex,
if i have ds1.connect(ds2).map(new coflatmapper())
then, i need data to stop flowing from ds1 until some data arrives in ds2.
Is that possible? I know that's a problematic situation as ds1 may get
overwhelmed with data if not processed quickly, but any work around would
really helpful.
I was thinking of busy waiting, but I doubt that, the 2 maps work in
different threads.
Regards
Biplob
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Way to hold execution of one of the map operator in Co-FlatMaps
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I might lead to flooding, yes. But I'm afraid it's the only way to go right
now.
Cheers,
Aljoscha
On Mon, 27 Jun 2016 at 17:57 Biplob Biswas <re...@gmail.com> wrote:
> Hi,
>
> I was afraid of buffering because I am not sure when the second map
> function
> would get data, wouldn't the first map be flooded with data while the
> second
> map gets data?
>
> Is it viable because the first stream is a stream of continuous data and
> the
> second stream gets data after some time and not as fast as the first one.
>
> Thanks for replying though, will try that out.
>
> Regards
> Biplob Biswas
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689p7699.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: Way to hold execution of one of the map operator in Co-FlatMaps
Posted by Biplob Biswas <re...@gmail.com>.
Hi,
I was afraid of buffering because I am not sure when the second map function
would get data, wouldn't the first map be flooded with data while the second
map gets data?
Is it viable because the first stream is a stream of continuous data and the
second stream gets data after some time and not as fast as the first one.
Thanks for replying though, will try that out.
Regards
Biplob Biswas
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689p7699.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Way to hold execution of one of the map operator in Co-FlatMaps
Posted by Aljoscha Krettek <al...@apache.org>.
Maybe. But how do you mean, exactly?
On Mon, 27 Jun 2016 at 11:14 Janardhan Reddy <ja...@olacabs.com>
wrote:
> Hi,
>
> Instead of buffering can we use event creation time and watermarks ?
>
> On Mon, Jun 27, 2016 at 2:32 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>> the two map functions are called by the same thread, so waiting in one
>> function would block all processing. What you could do is buffer elements
>> from one input and only process them when an element arrives on the other
>> input.
>>
>> Cheers,
>> Aljoscha
>>
>> On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <re...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I was wondering whether it is possible to stop streaming data in from
>>> one of
>>> the map operators until some data arrives in the second map operator.
>>>
>>> For ex,
>>>
>>> if i have ds1.connect(ds2).map(new coflatmapper())
>>>
>>> then, i need data to stop flowing from ds1 until some data arrives in
>>> ds2.
>>>
>>> Is that possible? I know that's a problematic situation as ds1 may get
>>> overwhelmed with data if not processed quickly, but any work around would
>>> really helpful.
>>>
>>>
>>> I was thinking of busy waiting, but I doubt that, the 2 maps work in
>>> different threads.
>>>
>>> Regards
>>> Biplob
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>
>
Re: Way to hold execution of one of the map operator in Co-FlatMaps
Posted by Janardhan Reddy <ja...@olacabs.com>.
Hi,
Instead of buffering can we use event creation time and watermarks ?
On Mon, Jun 27, 2016 at 2:32 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> the two map functions are called by the same thread, so waiting in one
> function would block all processing. What you could do is buffer elements
> from one input and only process them when an element arrives on the other
> input.
>
> Cheers,
> Aljoscha
>
> On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <re...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I was wondering whether it is possible to stop streaming data in from one
>> of
>> the map operators until some data arrives in the second map operator.
>>
>> For ex,
>>
>> if i have ds1.connect(ds2).map(new coflatmapper())
>>
>> then, i need data to stop flowing from ds1 until some data arrives in ds2.
>>
>> Is that possible? I know that's a problematic situation as ds1 may get
>> overwhelmed with data if not processed quickly, but any work around would
>> really helpful.
>>
>>
>> I was thinking of busy waiting, but I doubt that, the 2 maps work in
>> different threads.
>>
>> Regards
>> Biplob
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
Re: Way to hold execution of one of the map operator in Co-FlatMaps
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
the two map functions are called by the same thread, so waiting in one
function would block all processing. What you could do is buffer elements
from one input and only process them when an element arrives on the other
input.
Cheers,
Aljoscha
On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <re...@gmail.com> wrote:
> Hi,
>
> I was wondering whether it is possible to stop streaming data in from one
> of
> the map operators until some data arrives in the second map operator.
>
> For ex,
>
> if i have ds1.connect(ds2).map(new coflatmapper())
>
> then, i need data to stop flowing from ds1 until some data arrives in ds2.
>
> Is that possible? I know that's a problematic situation as ds1 may get
> overwhelmed with data if not processed quickly, but any work around would
> really helpful.
>
>
> I was thinking of busy waiting, but I doubt that, the 2 maps work in
> different threads.
>
> Regards
> Biplob
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>