You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Qing Lim <q....@mwam.com> on 2022/06/01 11:46:59 UTC

Can we resume a job from a savepoint from Java api?

Hi, is it possible to resume a job from a savepoint in Java code?

I wish to test failure recovery in my test code, I am thinking to simulate failure recovery by saving state to a save point and the recover from it, is this possible with local MiniCluster setup?

Kind regards

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.

To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.

To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. (“MWNA”), which is registered with the US Securities and Exchange Commission (“SEC”) as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.

Re: Can we resume a job from a savepoint from Java api?

Posted by Alexander Fedulov <al...@ververica.com>.

Hi Qing,

There are also some useful utilities in the Sink/Source testing suites that
you could either extract or adjust for your purposes [1]

[1]
https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/testsuites/SinkTestSuiteBase.java#L226

Best,
Alexander Fedulov

On Wed, Jun 1, 2022 at 3:33 PM Qing Lim <q....@mwam.com> wrote:

> Thanks both, that’s perfect!
>
>
>
> *From:* Jing Ge <ji...@ververica.com>
> *Sent:* 01 June 2022 14:29
> *To:* yuxia <lu...@alumni.sjtu.edu.cn>
> *Cc:* Qing Lim <q....@mwam.com>; User <us...@flink.apache.org>
> *Subject:* Re: Can we resume a job from a savepoint from Java api?
>
>
>
> Hi,
>
>
>
> yuxia has already pointed out the correct direction. The exact line for
> using the savepoint path to resume the job from a savepoint is at line 1326
> [1]
>
>
>
> [1]
> https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1326
>
>
>
>
>
> On Wed, Jun 1, 2022 at 3:13 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hope the unit
> test SavepointITCase#testCanRestoreWithModifiedStatelessOperators[1] in
> Flink repo  can help you.
>
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228
>
>
>
> Best regards,
> Yuxia
>
>
> ------------------------------
>
> *发件人**: *"Qing Lim" <q....@mwam.com>
> *收件人**: *"User" <us...@flink.apache.org>
> *发送时间**: *星期三, 2022年 6 月 01日 下午 7:46:59
> *主**题**: *Can we resume a job from a savepoint from Java api?
>
>
>
> Hi, is it possible to resume a job from a savepoint in Java code?
>
>
>
> I wish to test failure recovery in my test code, I am thinking to simulate
> failure recovery by saving state to a save point and the recover from it,
> is this possible with local MiniCluster setup?
>
>
>
> Kind regards
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser.  Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>
>
>
>

RE: Can we resume a job from a savepoint from Java api?

Posted by Qing Lim <q....@mwam.com>.

Thanks both, that’s perfect!

From: Jing Ge <ji...@ververica.com>
Sent: 01 June 2022 14:29
To: yuxia <lu...@alumni.sjtu.edu.cn>
Cc: Qing Lim <q....@mwam.com>; User <us...@flink.apache.org>
Subject: Re: Can we resume a job from a savepoint from Java api?

Hi,

yuxia has already pointed out the correct direction. The exact line for using the savepoint path to resume the job from a savepoint is at line 1326 [1]

[1] https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1326

On Wed, Jun 1, 2022 at 3:13 PM yuxia <lu...@alumni.sjtu.edu.cn>> wrote:
Hope the unit test SavepointITCase#testCanRestoreWithModifiedStatelessOperators[1] in Flink repo  can help you.

[1] https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228

Best regards,
Yuxia

________________________________
发件人: "Qing Lim" <q....@mwam.com>>
收件人: "User" <us...@flink.apache.org>>
发送时间: 星期三, 2022年 6 月 01日 下午 7:46:59
主题: Can we resume a job from a savepoint from Java api?

Hi, is it possible to resume a job from a savepoint in Java code?

I wish to test failure recovery in my test code, I am thinking to simulate failure recovery by saving state to a save point and the recover from it, is this possible with local MiniCluster setup?

Kind regards

This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.

To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.

To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser.  Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.

Re: Can we resume a job from a savepoint from Java api?

Posted by Jing Ge <ji...@ververica.com>.

Hi,

yuxia has already pointed out the correct direction. The exact line for
using the savepoint path to resume the job from a savepoint is at line 1326
[1]

[1]
https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1326


On Wed, Jun 1, 2022 at 3:13 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:

> Hope the unit
> test SavepointITCase#testCanRestoreWithModifiedStatelessOperators[1] in
> Flink repo  can help you.
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228
>
> Best regards,
> Yuxia
>
> ------------------------------
> *发件人: *"Qing Lim" <q....@mwam.com>
> *收件人: *"User" <us...@flink.apache.org>
> *发送时间: *星期三, 2022年 6 月 01日 下午 7:46:59
> *主题: *Can we resume a job from a savepoint from Java api?
>
> Hi, is it possible to resume a job from a savepoint in Java code?
>
>
>
> I wish to test failure recovery in my test code, I am thinking to simulate
> failure recovery by saving state to a save point and the recover from it,
> is this possible with local MiniCluster setup?
>
>
>
> Kind regards
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser.  Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>
>

Re: Can we resume a job from a savepoint from Java api?

Posted by yuxia <lu...@alumni.sjtu.edu.cn>.

Hope the unit test SavepointITCase#testCanRestoreWithModifiedStatelessOperators[1] in Flink repo can help you.

[1] [ https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228 | https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L1228 ]

Best regards,
Yuxia

发件人: "Qing Lim" <q....@mwam.com>
收件人: "User" <us...@flink.apache.org>
发送时间: 星期三, 2022年 6 月 01日 下午 7:46:59
主题: Can we resume a job from a savepoint from Java api?

Hi, is it possible to resume a job from a savepoint in Java code?

I wish to test failure recovery in my test code, I am thinking to simulate failure recovery by saving state to a save point and the recover from it, is this possible with local MiniCluster setup?

Kind regards

To find out more details about how we may collect, use and share your personal information, please see [ https://www.mwam.com/privacy-policy | https://www.mwam.com/privacy-policy ] . This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.

Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.