You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tomasz Dudziak <t....@mwam.com> on 2020/05/11 12:10:13 UTC
Flink REST API side effect?
Hi,
I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.
If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink's restart strategies are exhausted. If you don't and take a brief moment before your first query of the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.
Has anyone else noticed that behaviour?
Thanks,
Tomasz
Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: t.dudziak@mwam.com<ma...@mwam.com> | Tel: +44 207 024 7061
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy .
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.
Re: Flink REST API side effect?
Posted by Chesnay Schepler <ch...@apache.org>.
yes, that is correct.
On 11/05/2020 14:28, Tomasz Dudziak wrote:
>
> Thanks for reply. So do I understand correctly if I say that whenever
> you query job status it gets cached for a configurable amount of time
> and subsequent queries within that time period will not show any change?
>
> *From:*Chesnay Schepler <ch...@apache.org>
> *Sent:* 11 May 2020 13:20
> *To:* Tomasz Dudziak <t....@mwam.com>; user@flink.apache.org
> *Subject:* Re: Flink REST API side effect?
>
> This is expected, the backing data structure is cached for a while so
> we never hammer the JobManager with requests.
>
> IIRC this is controlled via "web.refresh-interval", with the default
> being 3 seconds.
>
> On 11/05/2020 14:10, Tomasz Dudziak wrote:
>
> Hi,
>
> I found an interesting behaviour of the REST API in my automated
> system tests using that API where I was getting status of a
> purposefully failing job.
>
> If you query job details immediately after job submission,
> subsequent queries will return its status as RUNNING for a moment
> until Flink’s restart strategies are exhausted. If you don’t and
> take a brief moment before your first query of the job status, it
> will be reported in a more fine-grained fashion, e.g. RESTARTING
> or FAILING.
>
> Has anyone else noticed that behaviour?
>
> Thanks,
>
> Tomasz
>
> *Tomasz Dudziak *| Marshall Wace LLP, George House, 131 Sloane
> Street, London, SW1X 9AT |**E-mail: t.dudziak@mwam.com
> <ma...@mwam.com>| Tel: +44 207 024 7061
>
> This e-mail and any attachments are confidential to the
> addressee(s) and may contain information that is legally
> privileged and/or confidential. Please refer to
> http://www.mwam.com/email-disclaimer-uk for important disclosures
> regarding this email. If we collect and use your personal data we
> will use it in accordance with our privacy policy, which can be
> reviewed at https://www.mwam.com/privacy-policy.
>
> Marshall Wace LLP is authorised and regulated by the Financial
> Conduct Authority. Marshall Wace LLP is a limited liability
> partnership registered in England and Wales with registered number
> OC302228 and registered office at George House, 131 Sloane Street,
> London, SW1X 9AT. If you are receiving this e-mail as a client, or
> an investor in an investment vehicle, managed or advised by
> Marshall Wace North America L.P., the sender of this e-mail is
> communicating with you in the sender's capacity as an associated
> or related person of Marshall Wace North America L.P., which is
> registered with the US Securities and Exchange Commission as an
> investment adviser.
>
> This e-mail and any attachments are confidential to the addressee(s)
> and may contain information that is legally privileged and/or
> confidential. Please refer to http://www.mwam.com/email-disclaimer-uk
> for important disclosures regarding this email. If we collect and use
> your personal data we will use it in accordance with our privacy
> policy, which can be reviewed at https://www.mwam.com/privacy-policy.
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership
> registered in England and Wales with registered number OC302228 and
> registered office at George House, 131 Sloane Street, London, SW1X
> 9AT. If you are receiving this e-mail as a client, or an investor in
> an investment vehicle, managed or advised by Marshall Wace North
> America L.P., the sender of this e-mail is communicating with you in
> the sender's capacity as an associated or related person of Marshall
> Wace North America L.P., which is registered with the US Securities
> and Exchange Commission as an investment adviser.
>
RE: Flink REST API side effect?
Posted by Tomasz Dudziak <t....@mwam.com>.
Thanks for reply. So do I understand correctly if I say that whenever you query job status it gets cached for a configurable amount of time and subsequent queries within that time period will not show any change?
From: Chesnay Schepler <ch...@apache.org>
Sent: 11 May 2020 13:20
To: Tomasz Dudziak <t....@mwam.com>; user@flink.apache.org
Subject: Re: Flink REST API side effect?
This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests.
IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds.
On 11/05/2020 14:10, Tomasz Dudziak wrote:
Hi,
I found an interesting behaviour of the REST API in my automated system tests using that API where I was getting status of a purposefully failing job.
If you query job details immediately after job submission, subsequent queries will return its status as RUNNING for a moment until Flink's restart strategies are exhausted. If you don't and take a brief moment before your first query of the job status, it will be reported in a more fine-grained fashion, e.g. RESTARTING or FAILING.
Has anyone else noticed that behaviour?
Thanks,
Tomasz
Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT | E-mail: t.dudziak@mwam.com<ma...@mwam.com> | Tel: +44 207 024 7061
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. Please refer to http://www.mwam.com/email-disclaimer-uk for important disclosures regarding this email. If we collect and use your personal data we will use it in accordance with our privacy policy, which can be reviewed at https://www.mwam.com/privacy-policy .
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P., which is registered with the US Securities and Exchange Commission as an investment adviser.
Re: Flink REST API side effect?
Posted by Chesnay Schepler <ch...@apache.org>.
This is expected, the backing data structure is cached for a while so we
never hammer the JobManager with requests.
IIRC this is controlled via "web.refresh-interval", with the default
being 3 seconds.
On 11/05/2020 14:10, Tomasz Dudziak wrote:
>
> Hi,
>
> I found an interesting behaviour of the REST API in my automated
> system tests using that API where I was getting status of a
> purposefully failing job.
>
> If you query job details immediately after job submission, subsequent
> queries will return its status as RUNNING for a moment until Flink’s
> restart strategies are exhausted. If you don’t and take a brief moment
> before your first query of the job status, it will be reported in a
> more fine-grained fashion, e.g. RESTARTING or FAILING.
>
> Has anyone else noticed that behaviour?
>
> Thanks,
>
> Tomasz
>
> *Tomasz Dudziak *| Marshall Wace LLP, George House, 131 Sloane Street,
> London, SW1X 9AT |**E-mail: t.dudziak@mwam.com
> <ma...@mwam.com>| Tel: +44 207 024 7061
>
> This e-mail and any attachments are confidential to the addressee(s)
> and may contain information that is legally privileged and/or
> confidential. Please refer to http://www.mwam.com/email-disclaimer-uk
> for important disclosures regarding this email. If we collect and use
> your personal data we will use it in accordance with our privacy
> policy, which can be reviewed at https://www.mwam.com/privacy-policy.
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership
> registered in England and Wales with registered number OC302228 and
> registered office at George House, 131 Sloane Street, London, SW1X
> 9AT. If you are receiving this e-mail as a client, or an investor in
> an investment vehicle, managed or advised by Marshall Wace North
> America L.P., the sender of this e-mail is communicating with you in
> the sender's capacity as an associated or related person of Marshall
> Wace North America L.P., which is registered with the US Securities
> and Exchange Commission as an investment adviser.
>