You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yitzchak Lieberman <yi...@sentinelone.com> on 2020/03/11 21:23:41 UTC
datadog http reporter metrics
Hi.
Did someone encountered problem with sending metrics with datadog http
reporter?
My setup is flink version 1.8.2 deployed on k8s with 1 job manager and 10
task managers.
Every version deploy I see metrics on my dashboard but after a few minutes
its stopped being sent from all task managers while job manager still sends
(with no error/warn on the logs).
Is it possible to be blocked by datadog due to the cluster size? my staging
cluster with 3 servers sends without any problem.
Thanks in advance,
Yitzchak.
Re: [EXT.MSG] Re: datadog http reporter metrics
Posted by Chesnay Schepler <ch...@apache.org>.
It would only be logged when using 1.10 unfortunately; but you should be
able to use the 1.10 version of the reporter with your version of Flink
to at least confirm that it is the same issue as FLINK-16611.
On 16/03/2020 11:35, Yitzchak Lieberman wrote:
> No, tried to find error/warn logs for rejected metrics, nothing...
> tor that case there should be an error, right? (when report is too large)
> I saw that there are some changes on version 1.10 for datadog
> reporter, maybe I should upgrade to this version?
>
> On Mon, Mar 16, 2020 at 11:47 AM Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> wrote:
>
> Do you see anything in the logs? In another thread a user reported
> that the datadog reporter could stop working when faced with a
> large number of metrics since datadog was rejecting the report due
> to being too large.
>
> On 15/03/2020 12:22, Yitzchak Lieberman wrote:
>> Anyone?
>>
>> On Wed, Mar 11, 2020 at 11:23 PM Yitzchak Lieberman
>> <yitzchakl@sentinelone.com <ma...@sentinelone.com>> wrote:
>>
>> Hi.
>>
>> Did someone encountered problem with sending metrics with
>> datadog http reporter?
>> My setup is flink version 1.8.2 deployed on k8s with 1 job
>> manager and 10 task managers.
>> Every version deploy I see metrics on my dashboard but after
>> a few minutes its stopped being sent from all task managers
>> while job manager still sends (with no error/warn on the logs).
>> Is it possible to be blocked by datadog due to the cluster
>> size? my staging cluster with 3 servers sends without any
>> problem.
>>
>> Thanks in advance,
>> Yitzchak.
>>
>
Re: [EXT.MSG] Re: datadog http reporter metrics
Posted by Yitzchak Lieberman <yi...@sentinelone.com>.
No, tried to find error/warn logs for rejected metrics, nothing...
tor that case there should be an error, right? (when report is too large)
I saw that there are some changes on version 1.10 for datadog reporter,
maybe I should upgrade to this version?
On Mon, Mar 16, 2020 at 11:47 AM Chesnay Schepler <ch...@apache.org>
wrote:
> Do you see anything in the logs? In another thread a user reported that
> the datadog reporter could stop working when faced with a large number of
> metrics since datadog was rejecting the report due to being too large.
>
> On 15/03/2020 12:22, Yitzchak Lieberman wrote:
>
> Anyone?
>
> On Wed, Mar 11, 2020 at 11:23 PM Yitzchak Lieberman <
> yitzchakl@sentinelone.com> wrote:
>
>> Hi.
>>
>> Did someone encountered problem with sending metrics with datadog http
>> reporter?
>> My setup is flink version 1.8.2 deployed on k8s with 1 job manager and 10
>> task managers.
>> Every version deploy I see metrics on my dashboard but after a few
>> minutes its stopped being sent from all task managers while job manager
>> still sends (with no error/warn on the logs).
>> Is it possible to be blocked by datadog due to the cluster size? my
>> staging cluster with 3 servers sends without any problem.
>>
>> Thanks in advance,
>> Yitzchak.
>>
>
>
Re: datadog http reporter metrics
Posted by Chesnay Schepler <ch...@apache.org>.
Do you see anything in the logs? In another thread a user reported that
the datadog reporter could stop working when faced with a large number
of metrics since datadog was rejecting the report due to being too large.
On 15/03/2020 12:22, Yitzchak Lieberman wrote:
> Anyone?
>
> On Wed, Mar 11, 2020 at 11:23 PM Yitzchak Lieberman
> <yitzchakl@sentinelone.com <ma...@sentinelone.com>> wrote:
>
> Hi.
>
> Did someone encountered problem with sending metrics with datadog
> http reporter?
> My setup is flink version 1.8.2 deployed on k8s with 1 job manager
> and 10 task managers.
> Every version deploy I see metrics on my dashboard but after a few
> minutes its stopped being sent from all task managers while job
> manager still sends (with no error/warn on the logs).
> Is it possible to be blocked by datadog due to the cluster size?
> my staging cluster with 3 servers sends without any problem.
>
> Thanks in advance,
> Yitzchak.
>
Re: datadog http reporter metrics
Posted by Yitzchak Lieberman <yi...@sentinelone.com>.
Anyone?
On Wed, Mar 11, 2020 at 11:23 PM Yitzchak Lieberman <
yitzchakl@sentinelone.com> wrote:
> Hi.
>
> Did someone encountered problem with sending metrics with datadog http
> reporter?
> My setup is flink version 1.8.2 deployed on k8s with 1 job manager and 10
> task managers.
> Every version deploy I see metrics on my dashboard but after a few minutes
> its stopped being sent from all task managers while job manager still sends
> (with no error/warn on the logs).
> Is it possible to be blocked by datadog due to the cluster size? my
> staging cluster with 3 servers sends without any problem.
>
> Thanks in advance,
> Yitzchak.
>