You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2017/10/12 14:33:44 UTC
Monitoring job w/LocalStreamEnvironment
Hi all,
With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.
I didn’t see support for async job creation, though.
So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.
But it appears like I need to have a ClusterClient in order to actually monitor this job.
And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.
Is there an easier/recommended approach to the above?
Thanks!
— Ken
--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378
Re: Monitoring job w/LocalStreamEnvironment
Posted by Piotr Nowojski <pi...@data-artisans.com>.
Hi,
Regarding metrics please check the "Writing an Integration test for flink-metrics” recent mailing list question. You can either use JMXReporter or write some custom reporter for this purpose.
Piotrek
> On 13 Oct 2017, at 20:57, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi Piotr,
>
> Thanks for responding, see below.
>
>> On Oct 12, 2017, at 7:51 AM, Piotr Nowojski <piotr@data-artisans.com <ma...@data-artisans.com>> wrote:
>>
>> Hi,
>>
>> Have you read the following doc?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html>
>>
>> There are some hints regarding testing your application. Especially take a look at the example with using static field to communicate with the running job.
>
> Yes, I’d read those.
>
> I already have a bunch of Flink metrics, I was hoping to leverage those to know when my test can safely terminate my iteration;
>
> I guess I could create a metrics wrapper that also logs to a static class during tests.
>
> Regards,
>
> — Ken
>
>
>>> On 12 Oct 2017, at 16:33, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>
>>> Hi all,
>>>
>>> With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.
>>>
>>> I didn’t see support for async job creation, though.
>>>
>>> So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.
>>>
>>> But it appears like I need to have a ClusterClient in order to actually monitor this job.
>>>
>>> And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.
>>>
>>> Is there an easier/recommended approach to the above?
>>>
>>> Thanks!
>>>
>>> — Ken
>>>
>>> --------------------------------------------
>>> http://about.me/kkrugler <http://about.me/kkrugler>
>>> +1 530-210-6378
>>>
>>
>
> --------------------------------------------
> http://about.me/kkrugler <http://about.me/kkrugler>
> +1 530-210-6378
Re: Monitoring job w/LocalStreamEnvironment
Posted by Piotr Nowojski <pi...@data-artisans.com>.
Hi,
Have you read the following doc?
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html>
There are some hints regarding testing your application. Especially take a look at the example with using static field to communicate with the running job.
Piotrek
> On 12 Oct 2017, at 16:33, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi all,
>
> With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.
>
> I didn’t see support for async job creation, though.
>
> So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.
>
> But it appears like I need to have a ClusterClient in order to actually monitor this job.
>
> And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.
>
> Is there an easier/recommended approach to the above?
>
> Thanks!
>
> — Ken
>
> --------------------------------------------
> http://about.me/kkrugler
> +1 530-210-6378
>