You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2018/01/04 14:52:01 UTC
Re: Service discovery for flink-metrics-prometheus
I'm not aware of how this is typically done but maybe Chesnay (cc'ed) has an idea.
> On 14. Dec 2017, at 16:55, Kien Truong <du...@gmail.com> wrote:
>
> Hi,
>
> Does anyone have recommendations about integrating flink-metrics-prometheus with some SD mechanism
>
> so that Prometheus can pick up the Task Manager's location dynamically ?
>
> Best regards,
>
> Kien
>
Re: Service discovery for flink-metrics-prometheus
Posted by Dongwon Kim <ea...@gmail.com>.
Hi all,
I also suffer from the lack of service discovery for flink-metrics-prometheus while using YARN for deployment, Prometheus for instrumentation, and Flink for stream processing.
I just upload a Python script for the purpose here: https://github.com/eastcirclek/flink-service-discovery <https://github.com/eastcirclek/flink-service-discovery>
Hope it can be helpful for your use case.
Best,
- Dongwon
https://github.com/eastcirclek/flink-service-discovery <https://github.com/eastcirclek/flink-service-discovery>
> 2018. 1. 8. 오후 7:27, Chesnay Schepler <ch...@apache.org> 작성:
>
> Yes, the logs are the only way to find out which port the reporter is bound to.
>
> We may be able to display this information in the web-UI, but it isn't very high on my list and will probably require
> modifications to the reporter interface.
>
> On 06.01.2018 04:24, Kien Truong wrote:
>> Hi,
>>
>> We are using YARN for deployment, so the combination of host&port for the Prometheus reporters can be really random depending on how the containers are co-located.
>>
>> One option we thought of was scrapping the log for this information, but it can be really messy in the long run.
>>
>> Regards,
>> Kien
>>
>> Sent from TypeApp <http://www.typeapp.com/r?b=11678>
>> On Jan 5, 2018, at 03:53, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>> How are you running deploying your Flink processes? For Service Discovery for Prometheus on Kubernetes, there are a few articles out there...
>>
>> On Thu, Jan 4, 2018 at 3:52 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
>> I'm not aware of how this is typically done but maybe Chesnay (cc'ed) has an idea.
>>
>> > On 14. Dec 2017, at 16:55, Kien Truong < duckientruong@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi,
>> >
>> > Does anyone have recommendations about integrating flink-metrics-prometheus with some SD mechanism
>> >
>> > so that Prometheus can pick up the Task Manager's location dynamically ?
>> >
>> > Best regards,
>> >
>> > Kien
>> >
>>
>>
>
Re: Service discovery for flink-metrics-prometheus
Posted by Chesnay Schepler <ch...@apache.org>.
Yes, the logs are the only way to find out which port the reporter is
bound to.
We may be able to display this information in the web-UI, but it isn't
very high on my list and will probably require
modifications to the reporter interface.
On 06.01.2018 04:24, Kien Truong wrote:
> Hi,
>
> We are using YARN for deployment, so the combination of host&port for
> the Prometheus reporters can be really random depending on how the
> containers are co-located.
>
> One option we thought of was scrapping the log for this information,
> but it can be really messy in the long run.
>
> Regards,
> Kien
>
> Sent from TypeApp <http://www.typeapp.com/r?b=11678>
> On Jan 5, 2018, at 03:53, Stephan Ewen <sewen@apache.org
> <ma...@apache.org>> wrote:
>
> How are you running deploying your Flink processes? For Service
> Discovery for Prometheus on Kubernetes, there are a few articles
> out there...
>
> On Thu, Jan 4, 2018 at 3:52 PM, Aljoscha Krettek
> <aljoscha@apache.org <ma...@apache.org>> wrote:
>
> I'm not aware of how this is typically done but maybe Chesnay
> (cc'ed) has an idea.
>
> > On 14. Dec 2017, at 16:55, Kien Truong <
> duckientruong@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi,
> >
> > Does anyone have recommendations about integrating
> flink-metrics-prometheus with some SD mechanism
> >
> > so that Prometheus can pick up the Task Manager's location
> dynamically ?
> >
> > Best regards,
> >
> > Kien
> >
>
>
Re: Service discovery for flink-metrics-prometheus
Posted by Kien Truong <du...@gmail.com>.
Hi,
We are using YARN for deployment, so the combination of host&port for the Prometheus reporters can be really random depending on how the containers are co-located.
One option we thought of was scrapping the log for this information, but it can be really messy in the long run.
Regards,
Kien
Sent from TypeApp
On Jan 5, 2018, 03:53, at 03:53, Stephan Ewen <se...@apache.org> wrote:
>How are you running deploying your Flink processes? For Service
>Discovery
>for Prometheus on Kubernetes, there are a few articles out there...
>
>On Thu, Jan 4, 2018 at 3:52 PM, Aljoscha Krettek <al...@apache.org>
>wrote:
>
>> I'm not aware of how this is typically done but maybe Chesnay (cc'ed)
>has
>> an idea.
>>
>> > On 14. Dec 2017, at 16:55, Kien Truong <du...@gmail.com>
>wrote:
>> >
>> > Hi,
>> >
>> > Does anyone have recommendations about integrating
>> flink-metrics-prometheus with some SD mechanism
>> >
>> > so that Prometheus can pick up the Task Manager's location
>dynamically ?
>> >
>> > Best regards,
>> >
>> > Kien
>> >
>>
>>
Re: Service discovery for flink-metrics-prometheus
Posted by Stephan Ewen <se...@apache.org>.
How are you running deploying your Flink processes? For Service Discovery
for Prometheus on Kubernetes, there are a few articles out there...
On Thu, Jan 4, 2018 at 3:52 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I'm not aware of how this is typically done but maybe Chesnay (cc'ed) has
> an idea.
>
> > On 14. Dec 2017, at 16:55, Kien Truong <du...@gmail.com> wrote:
> >
> > Hi,
> >
> > Does anyone have recommendations about integrating
> flink-metrics-prometheus with some SD mechanism
> >
> > so that Prometheus can pick up the Task Manager's location dynamically ?
> >
> > Best regards,
> >
> > Kien
> >
>
>