You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexander Filipchik <af...@gmail.com> on 2020/12/09 18:15:32 UTC
How to reduce number of metrics pushed to Prometheus Push Gateway
Hi,
Is there a way to reduce cardinality (preaggregate) metrics that are
emitted to Prom Push gateway?
Our metrics infra is struggling to digest per task stats. Any way we can
configure it to emit per stage aggregates?
Our current config:
metrics.scope.tm flink.taskmanager
metrics.scope.operator flink.operator
metrics.scope.jm flink.jobmanager
metrics.scope.jm.job flink.jobmanager.job
metrics.scope.task flink.task
But metrics still look like:
{ job_id="b12e2",
job_name="kafka_",
subtask_index="2",
task_attempt_num="14",
task_id="00f9d",
task_name="Source:_Read_from_kafka",
tm_id="17e3c"
}
Am I changing a proper config?
Alex
Re: How to reduce number of metrics pushed to Prometheus Push Gateway
Posted by Chesnay Schepler <ch...@apache.org>.
It will not produce aggregates. But it may reduce the load a bit without
affecting correctness; some variables are not necessarily required for
preventing metrics from overriding each other; like the job/task name
(because the IDs are good enough).
On 12/10/2020 6:37 PM, Alexander Filipchik wrote:
> Thank you for replying!
>
> Will exclusion produce proper aggregates? If I drop subtask_index,
> will resulting metric be a sum of all the subtasks, or it will be just
> data from one that was reported the last?
>
> Alex
>
> On Thu, Dec 10, 2020 at 4:28 AM Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> wrote:
>
> There is no way to reduce the number of metrics.
>
> The only thing you can do is exclude specific variables (e.g.,
> task_name), like this:
> metrics.reporter.<reporter_name>.scope.variables.exclude:task_name[;
> <any_other_variables_to_exclude]
>
> On 12/9/2020 7:15 PM, Alexander Filipchik wrote:
>> Hi,
>> Is there a way to reduce cardinality (preaggregate) metrics that
>> are emitted to Prom Push gateway?
>>
>> Our metrics infra is struggling to digest per task stats. Any way
>> we can configure it to emit per stage aggregates?
>>
>> Our current config:
>> metrics.scope.tm <http://metrics.scope.tm> flink.taskmanager
>> metrics.scope.operator flink.operator
>> metrics.scope.jm <http://metrics.scope.jm> flink.jobmanager
>> metrics.scope.jm.job flink.jobmanager.job
>> metrics.scope.task flink.task
>>
>>
>> But metrics still look like:
>> { job_id="b12e2",
>> job_name="kafka_",
>> subtask_index="2",
>> task_attempt_num="14",
>> task_id="00f9d",
>> task_name="Source:_Read_from_kafka",
>> tm_id="17e3c"
>> }
>>
>> Am I changing a proper config?
>>
>> Alex
>
>
Re: How to reduce number of metrics pushed to Prometheus Push Gateway
Posted by Alexander Filipchik <af...@gmail.com>.
Thank you for replying!
Will exclusion produce proper aggregates? If I drop subtask_index, will
resulting metric be a sum of all the subtasks, or it will be just data from
one that was reported the last?
Alex
On Thu, Dec 10, 2020 at 4:28 AM Chesnay Schepler <ch...@apache.org> wrote:
> There is no way to reduce the number of metrics.
>
> The only thing you can do is exclude specific variables (e.g., task_name),
> like this:
> metrics.reporter.<reporter_name>.scope.variables.exclude: task_name[;
> <any_other_variables_to_exclude]
>
> On 12/9/2020 7:15 PM, Alexander Filipchik wrote:
>
> Hi,
>
> Is there a way to reduce cardinality (preaggregate) metrics that are
> emitted to Prom Push gateway?
>
> Our metrics infra is struggling to digest per task stats. Any way we can
> configure it to emit per stage aggregates?
>
> Our current config:
> metrics.scope.tm flink.taskmanager
> metrics.scope.operator flink.operator
> metrics.scope.jm flink.jobmanager
> metrics.scope.jm.job flink.jobmanager.job
> metrics.scope.task flink.task
>
> But metrics still look like:
> { job_id="b12e2",
> job_name="kafka_",
> subtask_index="2",
> task_attempt_num="14",
> task_id="00f9d",
> task_name="Source:_Read_from_kafka",
> tm_id="17e3c"
> }
>
> Am I changing a proper config?
>
> Alex
>
>
>
Re: How to reduce number of metrics pushed to Prometheus Push Gateway
Posted by Chesnay Schepler <ch...@apache.org>.
There is no way to reduce the number of metrics.
The only thing you can do is exclude specific variables (e.g.,
task_name), like this:
metrics.reporter.<reporter_name>.scope.variables.exclude:task_name[;
<any_other_variables_to_exclude]
On 12/9/2020 7:15 PM, Alexander Filipchik wrote:
> Hi,
> Is there a way to reduce cardinality (preaggregate) metrics that are
> emitted to Prom Push gateway?
>
> Our metrics infra is struggling to digest per task stats. Any way we
> can configure it to emit per stage aggregates?
>
> Our current config:
> metrics.scope.tm <http://metrics.scope.tm> flink.taskmanager
> metrics.scope.operator flink.operator
> metrics.scope.jm <http://metrics.scope.jm> flink.jobmanager
> metrics.scope.jm.job flink.jobmanager.job
> metrics.scope.task flink.task
>
>
> But metrics still look like:
> { job_id="b12e2",
> job_name="kafka_",
> subtask_index="2",
> task_attempt_num="14",
> task_id="00f9d",
> task_name="Source:_Read_from_kafka",
> tm_id="17e3c"
> }
>
> Am I changing a proper config?
>
> Alex