You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Taher Koitawala <ta...@gslab.com> on 2019/01/10 08:29:00 UTC
SteamingFileSink with TwoPhaseCommit
Hi All,
As per my understanding and the API of StreamingFileSink,
TwoPhaseCommit is not being used. Can someone please confirm is that's
right? Also if StreamingFileSink does not support
TwoPhaseCommits what is the best way to implement this?
Regards,
Taher Koitawala
GS Lab Pune
+91 8407979163
Re: SteamingFileSink with TwoPhaseCommit
Posted by Taher Koitawala <ta...@gslab.com>.
Ok, thanks for the clarification. Really appreciate your help Kostas
On Thu 10 Jan, 2019, 6:19 PM Kostas Kloudas <k.kloudas@da-platform.com
wrote:
> Hi Taher,
>
> Well, I would say there is no single class that implements it.
> In a nutshell, it is the StreamingFileSink that (through Buckets) tells
> the responsible Bucket what to do at each step of the lifecycle of the
> Flink operator
> (mainly on element, on checkpoint, on checkpoint completed and on restore).
>
> So I would suggest that you should have a look in these classes.
> In essence, the main steps of the "choreography" are similar to the ones
> that the older BucketingSink was doing.
>
> Cheers,
> Kostas
>
> On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala <
> taher.koitawala@gslab.com> wrote:
>
>> Hi Kostas,
>> Thanks you for the clarification, also can you please
>> point how StreamingFileSink uses TwoPhaseCommit. Can you also point out the
>> implementing class for that?
>>
>>
>> Regards,
>> Taher Koitawala
>> GS Lab Pune
>> +91 8407979163
>>
>>
>> On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <kk...@gmail.com>
>> wrote:
>>
>>> That is correct.
>>> The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction
>>> because of some internal implementation details.
>>> But this, does not mean that it does not implement a two phase commit
>>> protocol (which is independent of the implementation).
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <
>>> taher.koitawala@gslab.com> wrote:
>>>
>>>> StreamingFileSink extends RichSinkFunction and implements
>>>> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however
>>>> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had
>>>> a question if the sink uses the TwoPhaseCommit protocol or not.
>>>>
>>>> Regards,
>>>> Taher Koitawala
>>>>
>>>> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kkloudas@gmail.com wrote:
>>>>
>>>>> Hi Taher,
>>>>>
>>>>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you
>>>>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being
>>>>> used"?
>>>>>
>>>>> Cheers,
>>>>> Kostas
>>>>>
>>>>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <
>>>>> taher.koitawala@gslab.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>> As per my understanding and the API of StreamingFileSink,
>>>>>> TwoPhaseCommit is not being used. Can someone please confirm is that's
>>>>>> right? Also if StreamingFileSink does not support
>>>>>> TwoPhaseCommits what is the best way to implement this?
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Taher Koitawala
>>>>>> GS Lab Pune
>>>>>> +91 8407979163
>>>>>>
>>>>>
Re: SteamingFileSink with TwoPhaseCommit
Posted by Kostas Kloudas <k....@da-platform.com>.
Hi Taher,
Well, I would say there is no single class that implements it.
In a nutshell, it is the StreamingFileSink that (through Buckets) tells the
responsible Bucket what to do at each step of the lifecycle of the Flink
operator
(mainly on element, on checkpoint, on checkpoint completed and on restore).
So I would suggest that you should have a look in these classes.
In essence, the main steps of the "choreography" are similar to the ones
that the older BucketingSink was doing.
Cheers,
Kostas
On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala <ta...@gslab.com>
wrote:
> Hi Kostas,
> Thanks you for the clarification, also can you please point
> how StreamingFileSink uses TwoPhaseCommit. Can you also point out the
> implementing class for that?
>
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>
>
> On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <kk...@gmail.com> wrote:
>
>> That is correct.
>> The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction
>> because of some internal implementation details.
>> But this, does not mean that it does not implement a two phase commit
>> protocol (which is independent of the implementation).
>>
>> Cheers,
>> Kostas
>>
>> On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <
>> taher.koitawala@gslab.com> wrote:
>>
>>> StreamingFileSink extends RichSinkFunction and implements
>>> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however
>>> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had
>>> a question if the sink uses the TwoPhaseCommit protocol or not.
>>>
>>> Regards,
>>> Taher Koitawala
>>>
>>> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kkloudas@gmail.com wrote:
>>>
>>>> Hi Taher,
>>>>
>>>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you
>>>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being
>>>> used"?
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <
>>>> taher.koitawala@gslab.com> wrote:
>>>>
>>>>> Hi All,
>>>>> As per my understanding and the API of StreamingFileSink,
>>>>> TwoPhaseCommit is not being used. Can someone please confirm is that's
>>>>> right? Also if StreamingFileSink does not support
>>>>> TwoPhaseCommits what is the best way to implement this?
>>>>>
>>>>>
>>>>> Regards,
>>>>> Taher Koitawala
>>>>> GS Lab Pune
>>>>> +91 8407979163
>>>>>
>>>>
Re: SteamingFileSink with TwoPhaseCommit
Posted by Taher Koitawala <ta...@gslab.com>.
Hi Kostas,
Thanks you for the clarification, also can you please point
how StreamingFileSink uses TwoPhaseCommit. Can you also point out the
implementing class for that?
Regards,
Taher Koitawala
GS Lab Pune
+91 8407979163
On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <kk...@gmail.com> wrote:
> That is correct.
> The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction
> because of some internal implementation details.
> But this, does not mean that it does not implement a two phase commit
> protocol (which is independent of the implementation).
>
> Cheers,
> Kostas
>
> On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <
> taher.koitawala@gslab.com> wrote:
>
>> StreamingFileSink extends RichSinkFunction and implements
>> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however
>> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had
>> a question if the sink uses the TwoPhaseCommit protocol or not.
>>
>> Regards,
>> Taher Koitawala
>>
>> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kkloudas@gmail.com wrote:
>>
>>> Hi Taher,
>>>
>>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you
>>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being
>>> used"?
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <
>>> taher.koitawala@gslab.com> wrote:
>>>
>>>> Hi All,
>>>> As per my understanding and the API of StreamingFileSink,
>>>> TwoPhaseCommit is not being used. Can someone please confirm is that's
>>>> right? Also if StreamingFileSink does not support
>>>> TwoPhaseCommits what is the best way to implement this?
>>>>
>>>>
>>>> Regards,
>>>> Taher Koitawala
>>>> GS Lab Pune
>>>> +91 8407979163
>>>>
>>>
Re: SteamingFileSink with TwoPhaseCommit
Posted by Kostas Kloudas <kk...@gmail.com>.
That is correct.
The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction
because of some internal implementation details.
But this, does not mean that it does not implement a two phase commit
protocol (which is independent of the implementation).
Cheers,
Kostas
On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala <ta...@gslab.com>
wrote:
> StreamingFileSink extends RichSinkFunction and implements
> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however
> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had
> a question if the sink uses the TwoPhaseCommit protocol or not.
>
> Regards,
> Taher Koitawala
>
> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kkloudas@gmail.com wrote:
>
>> Hi Taher,
>>
>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you
>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being used
>> "?
>>
>> Cheers,
>> Kostas
>>
>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <
>> taher.koitawala@gslab.com> wrote:
>>
>>> Hi All,
>>> As per my understanding and the API of StreamingFileSink,
>>> TwoPhaseCommit is not being used. Can someone please confirm is that's
>>> right? Also if StreamingFileSink does not support
>>> TwoPhaseCommits what is the best way to implement this?
>>>
>>>
>>> Regards,
>>> Taher Koitawala
>>> GS Lab Pune
>>> +91 8407979163
>>>
>>
Re: SteamingFileSink with TwoPhaseCommit
Posted by Taher Koitawala <ta...@gslab.com>.
StreamingFileSink extends RichSinkFunction and implements
CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however
TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had
a question if the sink uses the TwoPhaseCommit protocol or not.
Regards,
Taher Koitawala
On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kkloudas@gmail.com wrote:
> Hi Taher,
>
> The StreamingFileSink implements a version of TwoPhaseCommit. Can you
> elaborate a bit on what do you mean by " TwoPhaseCommit is not being used
> "?
>
> Cheers,
> Kostas
>
> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <ta...@gslab.com>
> wrote:
>
>> Hi All,
>> As per my understanding and the API of StreamingFileSink,
>> TwoPhaseCommit is not being used. Can someone please confirm is that's
>> right? Also if StreamingFileSink does not support
>> TwoPhaseCommits what is the best way to implement this?
>>
>>
>> Regards,
>> Taher Koitawala
>> GS Lab Pune
>> +91 8407979163
>>
>
Re: SteamingFileSink with TwoPhaseCommit
Posted by Kostas Kloudas <kk...@gmail.com>.
Hi Taher,
The StreamingFileSink implements a version of TwoPhaseCommit. Can you
elaborate a bit on what do you mean by " TwoPhaseCommit is not being used"?
Cheers,
Kostas
On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala <ta...@gslab.com>
wrote:
> Hi All,
> As per my understanding and the API of StreamingFileSink,
> TwoPhaseCommit is not being used. Can someone please confirm is that's
> right? Also if StreamingFileSink does not support
> TwoPhaseCommits what is the best way to implement this?
>
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>