You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Chan, Regina" <Re...@gs.com> on 2018/05/22 19:03:05 UTC

Data sink finishes before previous task finishes

Hi,

I'm still using Flink 1.2.1 on some production streams and I'm seeing this weird behavior where I'm seeing the flatMap stay in running while the data sink is in finished state. What's causing this behavior?


[cid:image001.png@01D3F1DD.F6FE3E40]


Regina Chan
Goldman Sachs - Enterprise Platforms, Data Architecture
30 Hudson Street, 37th floor | Jersey City, NY 07302 *  (212) 902-5697


________________________________

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>

Re: Data sink finishes before previous task finishes

Posted by Till Rohrmann <tr...@apache.org>.
Hi Regina,

from the top of my hat, I cannot tell why an intermediate operator stays in
state running while all others transitioned to a terminal state. Could you
share the logs with us? They might be helpful in understanding the problem
better.

Cheers,
Till

On Wed, May 23, 2018 at 12:44 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Hi,
>
> I don't know why this should happen. AFAIK, operators should transition to
> the finished state after their input operator finished.
> Canceling in case of a failure is a different thing, but obviously not the
> case here.
>
> This might be a bug.
> @Till, do you know what might cause such behavior in Flink 1.2.x?
>
> Best, Fabian
>
> 2018-05-22 21:03 GMT+02:00 Chan, Regina <Re...@gs.com>:
>
>> Hi,
>>
>>
>>
>> I’m still using Flink 1.2.1 on some production streams and I’m seeing
>> this weird behavior where I’m seeing the flatMap stay in running while the
>> data sink is in finished state. What’s causing this behavior?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Regina Chan*
>>
>> *Goldman Sachs* *–* Enterprise Platforms, Data Architecture
>>
>> *30 Hudson Street, 37th floor | Jersey City, NY 07302
>> <https://maps.google.com/?q=30+Hudson+Street,+37th+floor+%7C+Jersey+City,+NY+07302&entry=gmail&source=g>*
>> (  (212) 902-5697
>>
>>
>>
>> ------------------------------
>>
>> Your Personal Data: We may collect and process information about you that
>> may be subject to data protection laws. For more information about how we
>> use and disclose your personal data, how we protect your information, our
>> legal basis to use your information, your rights and who you can contact,
>> please refer to: www.gs.com/privacy-notices
>>
>
>

Re: Data sink finishes before previous task finishes

Posted by Fabian Hueske <fh...@gmail.com>.
Hi,

I don't know why this should happen. AFAIK, operators should transition to
the finished state after their input operator finished.
Canceling in case of a failure is a different thing, but obviously not the
case here.

This might be a bug.
@Till, do you know what might cause such behavior in Flink 1.2.x?

Best, Fabian

2018-05-22 21:03 GMT+02:00 Chan, Regina <Re...@gs.com>:

> Hi,
>
>
>
> I’m still using Flink 1.2.1 on some production streams and I’m seeing this
> weird behavior where I’m seeing the flatMap stay in running while the data
> sink is in finished state. What’s causing this behavior?
>
>
>
>
>
>
>
>
>
> *Regina Chan*
>
> *Goldman Sachs* *–* Enterprise Platforms, Data Architecture
>
> *30 Hudson Street, 37th floor | Jersey City, NY 07302
> <https://maps.google.com/?q=30+Hudson+Street,+37th+floor+%7C+Jersey+City,+NY+07302&entry=gmail&source=g>*
> (  (212) 902-5697
>
>
>
> ------------------------------
>
> Your Personal Data: We may collect and process information about you that
> may be subject to data protection laws. For more information about how we
> use and disclose your personal data, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to: www.gs.com/privacy-notices
>