You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by aryan m <ma...@gmail.com> on 2022/08/07 20:02:19 UTC
SDK Worker availability metrics
Hi Users!
Is there a recommended approach to publish metrics on the number of sdk
workers available/running as a gauge ?
[1]
https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267
[2]
https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148
-- Aryan
Re: SDK Worker availability metrics
Posted by aryan m <ma...@gmail.com>.
Hi Luke!
I agree `sdk_worker_parallelism`don't change after job submission.
However, users can change the configuration from m -> n over a period of
time.
Having this information as a metric helps in observing the
behavior/impact of the job with the config change.
[1]
https://github.com/apache/beam/blob/master/website/www/site/content/en/documentation/runtime/sdk-harness-config.md#sdk-harness-configuration
On Wed, Aug 10, 2022 at 1:05 PM Luke Cwik <lc...@google.com> wrote:
> Flink has a set of workers, each worker has a number of task slots. A
> pipeline will use the number of slots based upon what it was configured to
> run with.
>
> Are you trying to get the total number of workers, total number of tasks
> slots, number of task slots your pipeline is using or number of workers
> your pipeline is executing on?
>
> I was under the impression that the first two were properties of the Flink
> cluster and don't change while the third property is configured at job
> submission time and also doesn't change.
>
> I may not be understanding what you're trying to measure and why at
> pipeline runtime for Flink since many of these values don't change through
> the lifetime of the cluster and/or job.
>
> On Mon, Aug 8, 2022 at 4:59 PM aryan m <ma...@gmail.com> wrote:
>
>> Hi Luke!
>> Thanks !! We use the Flink Runner and run SDK workers as processes
>> [1] within a k8s pod. Can you please share broad steps on how one can do in
>> the runner ?
>>
>>
>> [1]
>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
>>
>>
>> On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user <us...@beam.apache.org>
>> wrote:
>>
>>> That code only executes within a runner and is only used by certain
>>> runners and wouldn't work in general from user code that is monitoring the
>>> job or user code executing within one of the workers.
>>>
>>> You would need to author code that is likely runner specific to look up
>>> the number of workers associated with a job as I don't believe there is a
>>> general way to do this for an arbitrary Apache Beam runner.
>>>
>>> Which runner would you most likely want to use?
>>>
>>> On Sun, Aug 7, 2022 at 1:02 PM aryan m <ma...@gmail.com> wrote:
>>>
>>>> Hi Users!
>>>> Is there a recommended approach to publish metrics on the number of
>>>> sdk workers available/running as a gauge ?
>>>>
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267
>>>> [2]
>>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148
>>>>
>>>>
>>>> -- Aryan
>>>>
>>>
Re: SDK Worker availability metrics
Posted by Luke Cwik via user <us...@beam.apache.org>.
Flink has a set of workers, each worker has a number of task slots. A
pipeline will use the number of slots based upon what it was configured to
run with.
Are you trying to get the total number of workers, total number of tasks
slots, number of task slots your pipeline is using or number of workers
your pipeline is executing on?
I was under the impression that the first two were properties of the Flink
cluster and don't change while the third property is configured at job
submission time and also doesn't change.
I may not be understanding what you're trying to measure and why at
pipeline runtime for Flink since many of these values don't change through
the lifetime of the cluster and/or job.
On Mon, Aug 8, 2022 at 4:59 PM aryan m <ma...@gmail.com> wrote:
> Hi Luke!
> Thanks !! We use the Flink Runner and run SDK workers as processes [1]
> within a k8s pod. Can you please share broad steps on how one can do in the
> runner ?
>
>
> [1]
> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
>
>
> On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user <us...@beam.apache.org>
> wrote:
>
>> That code only executes within a runner and is only used by certain
>> runners and wouldn't work in general from user code that is monitoring the
>> job or user code executing within one of the workers.
>>
>> You would need to author code that is likely runner specific to look up
>> the number of workers associated with a job as I don't believe there is a
>> general way to do this for an arbitrary Apache Beam runner.
>>
>> Which runner would you most likely want to use?
>>
>> On Sun, Aug 7, 2022 at 1:02 PM aryan m <ma...@gmail.com> wrote:
>>
>>> Hi Users!
>>> Is there a recommended approach to publish metrics on the number of
>>> sdk workers available/running as a gauge ?
>>>
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267
>>> [2]
>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148
>>>
>>>
>>> -- Aryan
>>>
>>
Re: SDK Worker availability metrics
Posted by aryan m <ma...@gmail.com>.
Hi Luke!
Thanks !! We use the Flink Runner and run SDK workers as processes [1]
within a k8s pod. Can you please share broad steps on how one can do in the
runner ?
[1]
https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user <us...@beam.apache.org>
wrote:
> That code only executes within a runner and is only used by certain
> runners and wouldn't work in general from user code that is monitoring the
> job or user code executing within one of the workers.
>
> You would need to author code that is likely runner specific to look up
> the number of workers associated with a job as I don't believe there is a
> general way to do this for an arbitrary Apache Beam runner.
>
> Which runner would you most likely want to use?
>
> On Sun, Aug 7, 2022 at 1:02 PM aryan m <ma...@gmail.com> wrote:
>
>> Hi Users!
>> Is there a recommended approach to publish metrics on the number of
>> sdk workers available/running as a gauge ?
>>
>>
>> [1]
>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267
>> [2]
>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148
>>
>>
>> -- Aryan
>>
>
Re: SDK Worker availability metrics
Posted by Luke Cwik via user <us...@beam.apache.org>.
That code only executes within a runner and is only used by certain runners
and wouldn't work in general from user code that is monitoring the job or
user code executing within one of the workers.
You would need to author code that is likely runner specific to look up the
number of workers associated with a job as I don't believe there is a
general way to do this for an arbitrary Apache Beam runner.
Which runner would you most likely want to use?
On Sun, Aug 7, 2022 at 1:02 PM aryan m <ma...@gmail.com> wrote:
> Hi Users!
> Is there a recommended approach to publish metrics on the number of
> sdk workers available/running as a gauge ?
>
>
> [1]
> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267
> [2]
> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148
>
>
> -- Aryan
>