You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vinay Patil <vi...@gmail.com> on 2016/11/07 19:13:54 UTC
Kafka Monitoring
Hi,
I am monitoring Kafka using KafkaManager for checking offset lag and other
Kafka metrics, however I am not able to see the consumers when I use
FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
list.
I have set the required parameters for the kafka consumer while running the
application.
Has anyone faced this issue ?
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Daniel,
My Zookeeper instances are running and I have provided the hosts in Kafka
manager config files.
Can you please elaborate how do I set it up with Zookeeper ?
I have also tried setting offset.storage as Zookeeper , still I don't see
the consumer getting listed in KafkaManager, also the kafka offset checker
is not showing the consumers.
Is there any configuration I am missing ?
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 2:32 AM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hello,
>
> I have been using that setup.
> From my understanding, if one desires to see the offset being consumed by
> Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will
> only serve as a view of progress.
>
> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
> purposes only.
>
> Best Regards,
> Daniel Santos
>
> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9961&i=0>> wrote:
>>
>> Hi,
>>
>> I am monitoring Kafka using KafkaManager for checking offset lag and
>> other Kafka metrics, however I am not able to see the consumers when I use
>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>> list.
>>
>> I have set the required parameters for the kafka consumer while running
>> the application.
>>
>> Has anyone faced this issue ?
>> I am using Kafka 0.9.0.1
>>
>> Regards,
>> Vinay Patil
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9961.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9963.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Daniel,
Yes Now I am able to see it , this was just a dummy code I was running on
local VM.
However on cluster , I had enabled checkpointing, still I was not able to
see the consumers, I guess I have to put the brokerPath as you have
provided in the properties.
From where did you get the brokerPath ? I mean for which property have you
set it
Regards,
Vinay Patil
On Wed, Nov 9, 2016 at 12:32 AM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hello,
>
> On flink do you have the checkpoint enabled ?
>
> env.enableCheckpointing(interval = CHKPOINT_INTERVAL)
>
> Regards,
>
> Daniel Santos
>
> On 11/08/2016 12:30 PM, vinay patil wrote:
>
> Yes Kafka and Flink connect to that zookeeper only.
>
> Not sure why it is not listing the consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9983&i=0>> wrote:
>
>> Hi,
>>
>> brokerPath is just optional.
>>
>> Used if you want to have multile kafka clusters.
>>
>> Each kafka cluster would connect to the same brokerPath.
>>
>> Since I have multiple clusters I use the brokerPath.
>>
>> From the looks of it you dont. So never mind it doesn't matter.
>>
>> You only have one zookeeper correct ?
>>
>> And kafka and flink connects to that only zookeeper ?
>>
>> Best Regards,
>>
>> Daniel Santos
>>
>> On 11/08/2016 11:18 AM, vinay patil wrote:
>>
>> Hi Daniel,
>>
>> Yes I have specified the zookeeper host in server.properties file , so
>> the broker is connected to zookeeper.
>>
>> https://kafka.apache.org/documentation#brokerconfigs -> according to
>> this link, I guess all these configs are done in server.prop , so from
>> where did you get kafka09 as brokerPath ?
>>
>> this is my entry in server.prop file -> zookeeper.connect=localhost:2181
>> Have you set this as zkhost:2181/kafka09 ?
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>>
>>> Hi,
>>>
>>> Your kafka broker is connected to zookeeper I believe.
>>>
>>> I am using kafka 0.9.0.1 my self too.
>>>
>>> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a
>>> path, for instances :
>>>
>>> zk1:2181,zk2:2181,zk3:2181/kafka09
>>>
>>> https://kafka.apache.org/documentation#brokerconfigs
>>>
>>> Now on the flink side I would configure "props.setProperty("zookeeper.connect",
>>> zkHosts)" the same resulting in :
>>>
>>>
>>> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ka
>>> fka09")
>>>
>>>
>>> That is what I mean by broker's path.
>>>
>>> Best Regards,
>>>
>>> Daniel Santos
>>>
>>> On 11/08/2016 10:49 AM, vinay patil wrote:
>>>
>>> Hi Daniel,
>>>
>>> I have the same properties set for the consumer and the same code
>>>
>>> *brokerspath only needed if you have set it on kafka config* -> I did
>>> not get this, do you mean to check the brokerspath in
>>> conf/server.properties file ? I have even tried by setting offset.storage
>>> property to zookeeeper, but still not getting the consumers listed
>>>
>>> I am using Kafka 0.9.0.1
>>>
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
>>> Mailing List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>>
>>>> Hello,
>>>>
>>>> This is my config.
>>>>
>>>> On kafka props :
>>>>
>>>> val props = new Properties()
>>>>
>>>> props.setProperty("zookeeper.connect", zkHosts)
>>>> props.setProperty("bootstrap.servers", kafHosts)
>>>> props.setProperty("group.id", "prod")
>>>> props.setProperty("auto.offset.reset", "earliest")
>>>>
>>>> Now for zkHosts beware that all your hosts quorum has to be included.
>>>>
>>>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>>>
>>>> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
>>>> .
>>>>
>>>> brokerspath only needed if you have set it on kafka config. Ignore it
>>>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>>>
>>>> After that -> val source = env.addSource(new
>>>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(),
>>>> props))
>>>>
>>>> Then on kafkamanager -> consumers I have the groupID prod.
>>>>
>>>> Hope it helps.
>>>>
>>>> Best Regards,
>>>>
>>>> Daniel Santos
>>>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>>>
>>>> Hi Limbo,
>>>>
>>>> I am using 0.9, I am not able to see updated results even after
>>>> refreshing.
>>>> There is some property that we have to set in order to make this work
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
>>>> List archive.] <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>>>
>>>>> I am using kafka 0.8, just refresh the page and you will see the
>>>>> updated results, it’s not auto update.
>>>>>
>>>>> There is our logstash consumer:
>>>>>
>>>>>
>>>>>
>>>>> and this is our flink consumer:
>>>>>
>>>>>
>>>>> I find the flink consumer just write the offset of the kafka partition
>>>>> to zookeeper without owner and ids,
>>>>> so we can’t find the consumer in the manager page, we can only find
>>>>> the offset info.
>>>>>
>>>>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>>>>>
>>>>> Hi Limbo,
>>>>>
>>>>> I can see the lag by using that URL, but the Lag there is not showing
>>>>> updated results, it does not change, also if you try to to change the
>>>>> consumer group value it will still show you the same value instead of
>>>>> saying consumer group does not exist or similar kind of error :)
>>>>>
>>>>> According to documentation of 0.9.x the offsets are stored in Kafka,
>>>>> but we can set offset.storage property to zookeeper.
>>>>> Even by setting this I am not getting the consumer listed.
>>>>>
>>>>> Kafka cli command also does not show this consumer
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing
>>>>> List archive.] <<a href="x-msg://4/user/SendEmail
>>>>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow"
>>>>> link="external" class="">[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have the same problem, I think the reason is that the consumer of
>>>>>> flink use the low level API,
>>>>>> and when I type the group name in manager url I can get the lag of
>>>>>> the flink consumer, like this:
>>>>>>
>>>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>>>>> sumer_name>/topic/<topic_name>/type/ZK
>>>>>>
>>>>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have been using that setup.
>>>>>> From my understanding, if one desires to see the offset being
>>>>>> consumed by Flink on KafkaManger, one has to set it up with zookeeper. On
>>>>>> 0.9 it will only serve as a view of progress.
>>>>>>
>>>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>>>>> purposes only.
>>>>>>
>>>>>> Best Regards,
>>>>>> Daniel Santos
>>>>>>
>>>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>>>>> other Kafka metrics, however I am not able to see the consumers when I use
>>>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>>>>> list.
>>>>>>>
>>>>>>> I have set the required parameters for the kafka consumer while
>>>>>>> running the application.
>>>>>>>
>>>>>>> Has anyone faced this issue ?
>>>>>>> I am using Kafka 0.9.0.1
>>>>>>>
>>>>>>> Regards,
>>>>>>> Vinay Patil
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> If you reply to this email, your message will be added to the
>>>>>> discussion below:
>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>>> email <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>>>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>>> here.
>>>>>> NAML
>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> View this message in context: Re: Kafka Monitoring
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>>> archive
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>> at Nabble.com <http://nabble.com/>.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> If you reply to this email, your message will be added to the
>>>>> discussion below:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>> ble.com/Kafka-Monitoring-tp9957p9965.html
>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>> email [hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>> here.
>>>>> NAML
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> View this message in context: Re: Kafka Monitoring
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>> at Nabble.com.
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Kafka-Monitoring-tp9957p9976.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9979.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9980&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9982.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9983&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9996.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p10006.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by Daniel Santos <ds...@cryptolab.net>.
Hello,
On flink do you have the checkpoint enabled ?
env.enableCheckpointing(interval = CHKPOINT_INTERVAL)
Regards,
Daniel Santos
On 11/08/2016 12:30 PM, vinay patil wrote:
> Yes Kafka and Flink connect to that zookeeper only.
>
> Not sure why it is not listing the consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> </user/SendEmail.jtp?type=node&node=9983&i=0>> wrote:
>
> Hi,
>
> brokerPath is just optional.
>
> Used if you want to have multile kafka clusters.
>
> Each kafka cluster would connect to the same brokerPath.
>
> Since I have multiple clusters I use the brokerPath.
>
> From the looks of it you dont. So never mind it doesn't matter.
>
> You only have one zookeeper correct ?
>
> And kafka and flink connects to that only zookeeper ?
>
> Best Regards,
>
> Daniel Santos
>
>
> On 11/08/2016 11:18 AM, vinay patil wrote:
>> Hi Daniel,
>>
>> Yes I have specified the zookeeper host in server.properties file
>> , so the broker is connected to zookeeper.
>>
>> https://kafka.apache.org/documentation#brokerconfigs
>> <https://kafka.apache.org/documentation#brokerconfigs> ->
>> according to this link, I guess all these configs are done in
>> server.prop , so from where did you get kafka09 as brokerPath ?
>>
>> this is my entry in server.prop file ->
>> zookeeper.connect=localhost:2181
>> Have you set this as zkhost:2181/kafka09 ?
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink
>> User Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>>
>> Hi,
>>
>> Your kafka broker is connected to zookeeper I believe.
>>
>> I am using kafka 0.9.0.1 my self too.
>>
>> On kafka broker 0.9.0.1 I have configured the zookeeper
>> connect to a path, for instances :
>>
>> zk1:2181,zk2:2181,zk3:2181/kafka09
>>
>> https://kafka.apache.org/documentation#brokerconfigs
>> <https://kafka.apache.org/documentation#brokerconfigs>
>>
>> Now on the flink side I would configure
>> "props.setProperty("zookeeper.connect", zkHosts)" the same
>> resulting in :
>>
>>
>> props.setProperty("zookeeperconnect",
>> "zk1:2181,zk2:2181,zk3:2181/kafka09")
>>
>>
>> That is what I mean by broker's path.
>>
>> Best Regards,
>>
>> Daniel Santos
>>
>>
>> On 11/08/2016 10:49 AM, vinay patil wrote:
>>> Hi Daniel,
>>>
>>> I have the same properties set for the consumer and the same
>>> code
>>>
>>> /brokerspath only needed if you have set it on kafka config/
>>> -> I did not get this, do you mean to check the brokerspath
>>> in conf/server.properties file ? I have even tried by
>>> setting offset.storage property to zookeeeper, but still not
>>> getting the consumers listed
>>>
>>> I am using Kafka 0.9.0.1
>>>
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache
>>> Flink User Mailing List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>>
>>> Hello,
>>>
>>> This is my config.
>>>
>>> On kafka props :
>>>
>>> val props = new Properties()
>>>
>>> props.setProperty("zookeeper.connect", zkHosts)
>>> props.setProperty("bootstrap.servers", kafHosts)
>>> props.setProperty("group.id <http://group.id>", "prod")
>>> props.setProperty("auto.offset.reset", "earliest")
>>>
>>> Now for zkHosts beware that all your hosts quorum has to
>>> be included.
>>>
>>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>>
>>> Then it will result in zkHosts being ->
>>> zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>>>
>>> brokerspath only needed if you have set it on kafka
>>> config. Ignore it otherwise, resulting in
>>> "zk1:2181,zk2:2181,zk3:2181" .
>>>
>>> After that -> val source = env.addSource(new
>>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
>>> SimpleStringSchema(), props))
>>>
>>> Then on kafkamanager -> consumers I have the groupID prod.
>>>
>>> Hope it helps.
>>>
>>> Best Regards,
>>>
>>> Daniel Santos
>>>
>>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>>> Hi Limbo,
>>>>
>>>> I am using 0.9, I am not able to see updated results
>>>> even after refreshing.
>>>> There is some property that we have to set in order to
>>>> make this work
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache
>>>> Flink User Mailing List archive.] <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>>
>>>> wrote:
>>>>
>>>> I am using kafka 0.8, just refresh the page and you
>>>> will see the updated results, it\u2019s not auto update.
>>>>
>>>> There is our logstash consumer:
>>>>
>>>>
>>>>
>>>> and this is our flink consumer:
>>>>
>>>>
>>>> I find the flink consumer just write the offset of
>>>> the kafka partition to zookeeper without owner and
>>>> ids,
>>>> so we can\u2019t find the consumer in the manager page,
>>>> we can only find the offset info.
>>>>
>>>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0b\u534812:02\uff0cvinay patil <[hidden
>>>>> email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
>>>>> \u5199\u9053\uff1a
>>>>>
>>>>> Hi Limbo,
>>>>>
>>>>> I can see the lag by using that URL, but the Lag
>>>>> there is not showing updated results, it does not
>>>>> change, also if you try to to change the consumer
>>>>> group value it will still show you the same value
>>>>> instead of saying consumer group does not exist or
>>>>> similar kind of error :)
>>>>>
>>>>> According to documentation of 0.9.x the offsets
>>>>> are stored in Kafka, but we can set offset.storage
>>>>> property to zookeeper.
>>>>> Even by setting this I am not getting the consumer
>>>>> listed.
>>>>>
>>>>> Kafka cli command also does not show this consumer
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache
>>>>> Flink User Mailing List archive.]<<a
>>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>>>>> target="_top" rel="nofollow" link="external"
>>>>> class="">[hidden email]>wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have the same problem, I think the reason is
>>>>> that the consumer of flink use the low level API,
>>>>> and when I type the group name in manager url
>>>>> I can get the lag of the flink consumer, like
>>>>> this:
>>>>>
>>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>>>> <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>>>
>>>>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0a\u53485:12\uff0cDaniel Santos
>>>>>> <[hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
>>>>>> \u5199\u9053\uff1a
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have been using that setup.
>>>>>> From my understanding, if one desires to see
>>>>>> the offset being consumed by Flink on
>>>>>> KafkaManger, one has to set it up with
>>>>>> zookeeper. On 0.9 it will only serve as a
>>>>>> view of progress.
>>>>>>
>>>>>> Basically what's mandatory on 0.8 is optional
>>>>>> on 0.9, and for viewing purposes only.
>>>>>>
>>>>>> Best Regards,
>>>>>> Daniel Santos
>>>>>>
>>>>>> On November 7, 2016 7:13:54 PM GMT+00:00,
>>>>>> Vinay Patil <[hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am monitoring Kafka using KafkaManager
>>>>>> for checking offset lag and other Kafka
>>>>>> metrics, however I am not able to see the
>>>>>> consumers when I use FlinkKafkaConsumer
>>>>>> , for console-consumer it shows them in
>>>>>> the Consumers list.
>>>>>>
>>>>>> I have set the required parameters for
>>>>>> the kafka consumer while running the
>>>>>> application.
>>>>>>
>>>>>> Has anyone faced this issue ?
>>>>>> I am using Kafka 0.9.0.1
>>>>>>
>>>>>> Regards,
>>>>>> Vinay Patil
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> If you reply to this email, your message will
>>>>> be added to the discussion below:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>>>> To start a new topic under Apache Flink User
>>>>> Mailing List archive., email<a
>>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>>>> target="_top" rel="nofollow" link="external"
>>>>> class="">[hidden email]
>>>>> To unsubscribe from Apache Flink User Mailing
>>>>> List archive.,click here.
>>>>> NAML
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> View this message in context:Re: Kafka Monitoring
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>>> Sent from theApache Flink User Mailing List
>>>>> archive. mailing list archive
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>>>> <http://nabble.com/>.
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> If you reply to this email, your message will be
>>>> added to the discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>>>
>>>> To start a new topic under Apache Flink User
>>>> Mailing List archive., email [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>> To unsubscribe from Apache Flink User Mailing List
>>>> archive., click here.
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> View this message in context: Re: Kafka Monitoring
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>>> Sent from the Apache Flink User Mailing List archive.
>>>> mailing list archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>> at Nabble.com.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> If you reply to this email, your message will be added
>>> to the discussion below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>>>
>>> To start a new topic under Apache Flink User Mailing
>>> List archive., email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>> To unsubscribe from Apache Flink User Mailing List
>>> archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>>> Sent from the Apache Flink User Mailing List archive.
>>> mailing list archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html>
>>
>> To start a new topic under Apache Flink User Mailing List
>> archive., email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9980&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive.,
>> click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
>> Sent from the Apache Flink User Mailing List archive. mailing
>> list archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9982.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9982.html>
>
> To start a new topic under Apache Flink User Mailing List
> archive., email [hidden email]
> </user/SendEmail.jtp?type=node&node=9983&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html>
> Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> at Nabble.com.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Yes Kafka and Flink connect to that zookeeper only.
Not sure why it is not listing the consumer
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi,
>
> brokerPath is just optional.
>
> Used if you want to have multile kafka clusters.
>
> Each kafka cluster would connect to the same brokerPath.
>
> Since I have multiple clusters I use the brokerPath.
>
> From the looks of it you dont. So never mind it doesn't matter.
>
> You only have one zookeeper correct ?
>
> And kafka and flink connects to that only zookeeper ?
>
> Best Regards,
>
> Daniel Santos
>
> On 11/08/2016 11:18 AM, vinay patil wrote:
>
> Hi Daniel,
>
> Yes I have specified the zookeeper host in server.properties file , so the
> broker is connected to zookeeper.
>
> https://kafka.apache.org/documentation#brokerconfigs -> according to
> this link, I guess all these configs are done in server.prop , so from
> where did you get kafka09 as brokerPath ?
>
> this is my entry in server.prop file -> zookeeper.connect=localhost:2181
> Have you set this as zkhost:2181/kafka09 ?
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>
>> Hi,
>>
>> Your kafka broker is connected to zookeeper I believe.
>>
>> I am using kafka 0.9.0.1 my self too.
>>
>> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a
>> path, for instances :
>>
>> zk1:2181,zk2:2181,zk3:2181/kafka09
>>
>> https://kafka.apache.org/documentation#brokerconfigs
>>
>> Now on the flink side I would configure "props.setProperty("zookeeper.connect",
>> zkHosts)" the same resulting in :
>>
>>
>> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ka
>> fka09")
>>
>>
>> That is what I mean by broker's path.
>>
>> Best Regards,
>>
>> Daniel Santos
>>
>> On 11/08/2016 10:49 AM, vinay patil wrote:
>>
>> Hi Daniel,
>>
>> I have the same properties set for the consumer and the same code
>>
>> *brokerspath only needed if you have set it on kafka config* -> I did
>> not get this, do you mean to check the brokerspath in
>> conf/server.properties file ? I have even tried by setting offset.storage
>> property to zookeeeper, but still not getting the consumers listed
>>
>> I am using Kafka 0.9.0.1
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>
>>> Hello,
>>>
>>> This is my config.
>>>
>>> On kafka props :
>>>
>>> val props = new Properties()
>>>
>>> props.setProperty("zookeeper.connect", zkHosts)
>>> props.setProperty("bootstrap.servers", kafHosts)
>>> props.setProperty("group.id", "prod")
>>> props.setProperty("auto.offset.reset", "earliest")
>>>
>>> Now for zkHosts beware that all your hosts quorum has to be included.
>>>
>>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>>
>>> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
>>> .
>>>
>>> brokerspath only needed if you have set it on kafka config. Ignore it
>>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>>
>>> After that -> val source = env.addSource(new
>>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(),
>>> props))
>>>
>>> Then on kafkamanager -> consumers I have the groupID prod.
>>>
>>> Hope it helps.
>>>
>>> Best Regards,
>>>
>>> Daniel Santos
>>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>>
>>> Hi Limbo,
>>>
>>> I am using 0.9, I am not able to see updated results even after
>>> refreshing.
>>> There is some property that we have to set in order to make this work
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
>>> List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>>
>>>> I am using kafka 0.8, just refresh the page and you will see the
>>>> updated results, it’s not auto update.
>>>>
>>>> There is our logstash consumer:
>>>>
>>>>
>>>>
>>>> and this is our flink consumer:
>>>>
>>>>
>>>> I find the flink consumer just write the offset of the kafka partition
>>>> to zookeeper without owner and ids,
>>>> so we can’t find the consumer in the manager page, we can only find the
>>>> offset info.
>>>>
>>>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>>>>
>>>> Hi Limbo,
>>>>
>>>> I can see the lag by using that URL, but the Lag there is not showing
>>>> updated results, it does not change, also if you try to to change the
>>>> consumer group value it will still show you the same value instead of
>>>> saying consumer group does not exist or similar kind of error :)
>>>>
>>>> According to documentation of 0.9.x the offsets are stored in Kafka,
>>>> but we can set offset.storage property to zookeeper.
>>>> Even by setting this I am not getting the consumer listed.
>>>>
>>>> Kafka cli command also does not show this consumer
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing
>>>> List archive.] <<a href="x-msg://4/user/SendEmail
>>>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow"
>>>> link="external" class="">[hidden email]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have the same problem, I think the reason is that the consumer of
>>>>> flink use the low level API,
>>>>> and when I type the group name in manager url I can get the lag of the
>>>>> flink consumer, like this:
>>>>>
>>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>>>> sumer_name>/topic/<topic_name>/type/ZK
>>>>>
>>>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have been using that setup.
>>>>> From my understanding, if one desires to see the offset being consumed
>>>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it
>>>>> will only serve as a view of progress.
>>>>>
>>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>>>> purposes only.
>>>>>
>>>>> Best Regards,
>>>>> Daniel Santos
>>>>>
>>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>>>> other Kafka metrics, however I am not able to see the consumers when I use
>>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>>>> list.
>>>>>>
>>>>>> I have set the required parameters for the kafka consumer while
>>>>>> running the application.
>>>>>>
>>>>>> Has anyone faced this issue ?
>>>>>> I am using Kafka 0.9.0.1
>>>>>>
>>>>>> Regards,
>>>>>> Vinay Patil
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> If you reply to this email, your message will be added to the
>>>>> discussion below:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>> email <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>> here.
>>>>> NAML
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> View this message in context: Re: Kafka Monitoring
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>> at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Kafka-Monitoring-tp9957p9965.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9976.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9979.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9980&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9982.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by Daniel Santos <ds...@cryptolab.net>.
Hi,
brokerPath is just optional.
Used if you want to have multile kafka clusters.
Each kafka cluster would connect to the same brokerPath.
Since I have multiple clusters I use the brokerPath.
From the looks of it you dont. So never mind it doesn't matter.
You only have one zookeeper correct ?
And kafka and flink connects to that only zookeeper ?
Best Regards,
Daniel Santos
On 11/08/2016 11:18 AM, vinay patil wrote:
> Hi Daniel,
>
> Yes I have specified the zookeeper host in server.properties file , so
> the broker is connected to zookeeper.
>
> https://kafka.apache.org/documentation#brokerconfigs
> <https://kafka.apache.org/documentation#brokerconfigs> -> according to
> this link, I guess all these configs are done in server.prop , so from
> where did you get kafka09 as brokerPath ?
>
> this is my entry in server.prop file -> zookeeper.connect=localhost:2181
> Have you set this as zkhost:2181/kafka09 ?
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> </user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>
> Hi,
>
> Your kafka broker is connected to zookeeper I believe.
>
> I am using kafka 0.9.0.1 my self too.
>
> On kafka broker 0.9.0.1 I have configured the zookeeper connect to
> a path, for instances :
>
> zk1:2181,zk2:2181,zk3:2181/kafka09
>
> https://kafka.apache.org/documentation#brokerconfigs
> <https://kafka.apache.org/documentation#brokerconfigs>
>
> Now on the flink side I would configure
> "props.setProperty("zookeeper.connect", zkHosts)" the same
> resulting in :
>
>
> props.setProperty("zookeeperconnect",
> "zk1:2181,zk2:2181,zk3:2181/kafka09")
>
>
> That is what I mean by broker's path.
>
> Best Regards,
>
> Daniel Santos
>
>
> On 11/08/2016 10:49 AM, vinay patil wrote:
>> Hi Daniel,
>>
>> I have the same properties set for the consumer and the same code
>>
>> /brokerspath only needed if you have set it on kafka config/ -> I
>> did not get this, do you mean to check the brokerspath in
>> conf/server.properties file ? I have even tried by setting
>> offset.storage property to zookeeeper, but still not getting the
>> consumers listed
>>
>> I am using Kafka 0.9.0.1
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink
>> User Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>
>> Hello,
>>
>> This is my config.
>>
>> On kafka props :
>>
>> val props = new Properties()
>>
>> props.setProperty("zookeeper.connect", zkHosts)
>> props.setProperty("bootstrap.servers", kafHosts)
>> props.setProperty("group.id <http://group.id>", "prod")
>> props.setProperty("auto.offset.reset", "earliest")
>>
>> Now for zkHosts beware that all your hosts quorum has to be
>> included.
>>
>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>
>> Then it will result in zkHosts being ->
>> zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>>
>> brokerspath only needed if you have set it on kafka config.
>> Ignore it otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>
>> After that -> val source = env.addSource(new
>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
>> SimpleStringSchema(), props))
>>
>> Then on kafkamanager -> consumers I have the groupID prod.
>>
>> Hope it helps.
>>
>> Best Regards,
>>
>> Daniel Santos
>>
>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>> Hi Limbo,
>>>
>>> I am using 0.9, I am not able to see updated results even
>>> after refreshing.
>>> There is some property that we have to set in order to make
>>> this work
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink
>>> User Mailing List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>>
>>> I am using kafka 0.8, just refresh the page and you will
>>> see the updated results, it\u2019s not auto update.
>>>
>>> There is our logstash consumer:
>>>
>>>
>>>
>>> and this is our flink consumer:
>>>
>>>
>>> I find the flink consumer just write the offset of the
>>> kafka partition to zookeeper without owner and ids,
>>> so we can\u2019t find the consumer in the manager page, we
>>> can only find the offset info.
>>>
>>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0b\u534812:02\uff0cvinay patil <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
>>>> \u5199\u9053\uff1a
>>>>
>>>> Hi Limbo,
>>>>
>>>> I can see the lag by using that URL, but the Lag there
>>>> is not showing updated results, it does not change,
>>>> also if you try to to change the consumer group value
>>>> it will still show you the same value instead of saying
>>>> consumer group does not exist or similar kind of error :)
>>>>
>>>> According to documentation of 0.9.x the offsets are
>>>> stored in Kafka, but we can set offset.storage property
>>>> to zookeeper.
>>>> Even by setting this I am not getting the consumer listed.
>>>>
>>>> Kafka cli command also does not show this consumer
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink
>>>> User Mailing List archive.]<<a
>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>>>> target="_top" rel="nofollow" link="external"
>>>> class="">[hidden email]>wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have the same problem, I think the reason is that
>>>> the consumer of flink use the low level API,
>>>> and when I type the group name in manager url I can
>>>> get the lag of the flink consumer, like this:
>>>>
>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>>> <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>>
>>>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0a\u53485:12\uff0cDaniel Santos <[hidden
>>>>> email]
>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
>>>>> \u5199\u9053\uff1a
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have been using that setup.
>>>>> From my understanding, if one desires to see the
>>>>> offset being consumed by Flink on KafkaManger, one
>>>>> has to set it up with zookeeper. On 0.9 it will
>>>>> only serve as a view of progress.
>>>>>
>>>>> Basically what's mandatory on 0.8 is optional on
>>>>> 0.9, and for viewing purposes only.
>>>>>
>>>>> Best Regards,
>>>>> Daniel Santos
>>>>>
>>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay
>>>>> Patil <[hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am monitoring Kafka using KafkaManager for
>>>>> checking offset lag and other Kafka metrics,
>>>>> however I am not able to see the consumers
>>>>> when I use FlinkKafkaConsumer , for
>>>>> console-consumer it shows them in the
>>>>> Consumers list.
>>>>>
>>>>> I have set the required parameters for the
>>>>> kafka consumer while running the application.
>>>>>
>>>>> Has anyone faced this issue ?
>>>>> I am using Kafka 0.9.0.1
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> If you reply to this email, your message will be
>>>> added to the discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>>> To start a new topic under Apache Flink User
>>>> Mailing List archive., email<a
>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>>> target="_top" rel="nofollow" link="external"
>>>> class="">[hidden email]
>>>> To unsubscribe from Apache Flink User Mailing List
>>>> archive.,click here.
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> View this message in context:Re: Kafka Monitoring
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>> Sent from theApache Flink User Mailing List archive.
>>>> mailing list archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>>> <http://nabble.com/>.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> If you reply to this email, your message will be added
>>> to the discussion below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>>
>>> To start a new topic under Apache Flink User Mailing
>>> List archive., email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>> To unsubscribe from Apache Flink User Mailing List
>>> archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>> Sent from the Apache Flink User Mailing List archive.
>>> mailing list archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>>
>> To start a new topic under Apache Flink User Mailing List
>> archive., email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive.,
>> click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>> Sent from the Apache Flink User Mailing List archive. mailing
>> list archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html>
>
> To start a new topic under Apache Flink User Mailing List
> archive., email [hidden email]
> </user/SendEmail.jtp?type=node&node=9980&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
> Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> at Nabble.com.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Daniel,
Yes I have specified the zookeeper host in server.properties file , so the
broker is connected to zookeeper.
https://kafka.apache.org/documentation#brokerconfigs -> according to this
link, I guess all these configs are done in server.prop , so from where did
you get kafka09 as brokerPath ?
this is my entry in server.prop file -> zookeeper.connect=localhost:2181
Have you set this as zkhost:2181/kafka09 ?
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi,
>
> Your kafka broker is connected to zookeeper I believe.
>
> I am using kafka 0.9.0.1 my self too.
>
> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a path,
> for instances :
>
> zk1:2181,zk2:2181,zk3:2181/kafka09
>
> https://kafka.apache.org/documentation#brokerconfigs
>
> Now on the flink side I would configure "props.setProperty("zookeeper.connect",
> zkHosts)" the same resulting in :
>
>
> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/
> kafka09")
>
>
> That is what I mean by broker's path.
>
> Best Regards,
>
> Daniel Santos
>
> On 11/08/2016 10:49 AM, vinay patil wrote:
>
> Hi Daniel,
>
> I have the same properties set for the consumer and the same code
>
> *brokerspath only needed if you have set it on kafka config* -> I did not
> get this, do you mean to check the brokerspath in conf/server.properties
> file ? I have even tried by setting offset.storage property to zookeeeper,
> but still not getting the consumers listed
>
> I am using Kafka 0.9.0.1
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>
>> Hello,
>>
>> This is my config.
>>
>> On kafka props :
>>
>> val props = new Properties()
>>
>> props.setProperty("zookeeper.connect", zkHosts)
>> props.setProperty("bootstrap.servers", kafHosts)
>> props.setProperty("group.id", "prod")
>> props.setProperty("auto.offset.reset", "earliest")
>>
>> Now for zkHosts beware that all your hosts quorum has to be included.
>>
>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>
>> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
>> .
>>
>> brokerspath only needed if you have set it on kafka config. Ignore it
>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>
>> After that -> val source = env.addSource(new
>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(),
>> props))
>>
>> Then on kafkamanager -> consumers I have the groupID prod.
>>
>> Hope it helps.
>>
>> Best Regards,
>>
>> Daniel Santos
>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>
>> Hi Limbo,
>>
>> I am using 0.9, I am not able to see updated results even after
>> refreshing.
>> There is some property that we have to set in order to make this work
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
>> List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>
>>> I am using kafka 0.8, just refresh the page and you will see the updated
>>> results, it’s not auto update.
>>>
>>> There is our logstash consumer:
>>>
>>>
>>>
>>> and this is our flink consumer:
>>>
>>>
>>> I find the flink consumer just write the offset of the kafka partition
>>> to zookeeper without owner and ids,
>>> so we can’t find the consumer in the manager page, we can only find the
>>> offset info.
>>>
>>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>>>
>>> Hi Limbo,
>>>
>>> I can see the lag by using that URL, but the Lag there is not showing
>>> updated results, it does not change, also if you try to to change the
>>> consumer group value it will still show you the same value instead of
>>> saying consumer group does not exist or similar kind of error :)
>>>
>>> According to documentation of 0.9.x the offsets are stored in Kafka, but
>>> we can set offset.storage property to zookeeper.
>>> Even by setting this I am not getting the consumer listed.
>>>
>>> Kafka cli command also does not show this consumer
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing
>>> List archive.] <<a href="x-msg://4/user/SendEmail
>>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow"
>>> link="external" class="">[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have the same problem, I think the reason is that the consumer of
>>>> flink use the low level API,
>>>> and when I type the group name in manager url I can get the lag of the
>>>> flink consumer, like this:
>>>>
>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>>> sumer_name>/topic/<topic_name>/type/ZK
>>>>
>>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>>>
>>>> Hello,
>>>>
>>>> I have been using that setup.
>>>> From my understanding, if one desires to see the offset being consumed
>>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it
>>>> will only serve as a view of progress.
>>>>
>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>>> purposes only.
>>>>
>>>> Best Regards,
>>>> Daniel Santos
>>>>
>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>>> other Kafka metrics, however I am not able to see the consumers when I use
>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>>> list.
>>>>>
>>>>> I have set the required parameters for the kafka consumer while
>>>>> running the application.
>>>>>
>>>>> Has anyone faced this issue ?
>>>>> I am using Kafka 0.9.0.1
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com <http://nabble.com/>.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9965.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9976.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9979.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by Daniel Santos <ds...@cryptolab.net>.
Hi,
Your kafka broker is connected to zookeeper I believe.
I am using kafka 0.9.0.1 my self too.
On kafka broker 0.9.0.1 I have configured the zookeeper connect to a
path, for instances :
zk1:2181,zk2:2181,zk3:2181/kafka09
https://kafka.apache.org/documentation#brokerconfigs
Now on the flink side I would configure
"props.setProperty("zookeeper.connect", zkHosts)" the same resulting in :
props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/kafka09")
That is what I mean by broker's path.
Best Regards,
Daniel Santos
On 11/08/2016 10:49 AM, vinay patil wrote:
> Hi Daniel,
>
> I have the same properties set for the consumer and the same code
>
> /brokerspath only needed if you have set it on kafka config/ -> I did
> not get this, do you mean to check the brokerspath in
> conf/server.properties file ? I have even tried by setting
> offset.storage property to zookeeeper, but still not getting the
> consumers listed
>
> I am using Kafka 0.9.0.1
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> </user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>
> Hello,
>
> This is my config.
>
> On kafka props :
>
> val props = new Properties()
>
> props.setProperty("zookeeper.connect", zkHosts)
> props.setProperty("bootstrap.servers", kafHosts)
> props.setProperty("group.id <http://group.id>", "prod")
> props.setProperty("auto.offset.reset", "earliest")
>
> Now for zkHosts beware that all your hosts quorum has to be included.
>
> For instances you have zk1 and zk2 and zk3 to form a quorum.
>
> Then it will result in zkHosts being ->
> zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>
> brokerspath only needed if you have set it on kafka config. Ignore
> it otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>
> After that -> val source = env.addSource(new
> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
> SimpleStringSchema(), props))
>
> Then on kafkamanager -> consumers I have the groupID prod.
>
> Hope it helps.
>
> Best Regards,
>
> Daniel Santos
>
> On 11/08/2016 08:45 AM, vinay patil wrote:
>> Hi Limbo,
>>
>> I am using 0.9, I am not able to see updated results even after
>> refreshing.
>> There is some property that we have to set in order to make this work
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>
>> I am using kafka 0.8, just refresh the page and you will see
>> the updated results, it\u2019s not auto update.
>>
>> There is our logstash consumer:
>>
>>
>>
>> and this is our flink consumer:
>>
>>
>> I find the flink consumer just write the offset of the kafka
>> partition to zookeeper without owner and ids,
>> so we can\u2019t find the consumer in the manager page, we can
>> only find the offset info.
>>
>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0b\u534812:02\uff0cvinay patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> \u5199\u9053\uff1a
>>>
>>> Hi Limbo,
>>>
>>> I can see the lag by using that URL, but the Lag there is
>>> not showing updated results, it does not change, also if you
>>> try to to change the consumer group value it will still show
>>> you the same value instead of saying consumer group does not
>>> exist or similar kind of error :)
>>>
>>> According to documentation of 0.9.x the offsets are stored
>>> in Kafka, but we can set offset.storage property to zookeeper.
>>> Even by setting this I am not getting the consumer listed.
>>>
>>> Kafka cli command also does not show this consumer
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User
>>> Mailing List archive.]<<a
>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>>> target="_top" rel="nofollow" link="external"
>>> class="">[hidden email]>wrote:
>>>
>>> Hi,
>>>
>>> I have the same problem, I think the reason is that the
>>> consumer of flink use the low level API,
>>> and when I type the group name in manager url I can get
>>> the lag of the flink consumer, like this:
>>>
>>> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>> <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>
>>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0a\u53485:12\uff0cDaniel Santos <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> \u5199\u9053\uff1a
>>>>
>>>> Hello,
>>>>
>>>> I have been using that setup.
>>>> From my understanding, if one desires to see the offset
>>>> being consumed by Flink on KafkaManger, one has to set
>>>> it up with zookeeper. On 0.9 it will only serve as a
>>>> view of progress.
>>>>
>>>> Basically what's mandatory on 0.8 is optional on 0.9,
>>>> and for viewing purposes only.
>>>>
>>>> Best Regards,
>>>> Daniel Santos
>>>>
>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil
>>>> <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am monitoring Kafka using KafkaManager for
>>>> checking offset lag and other Kafka metrics,
>>>> however I am not able to see the consumers when I
>>>> use FlinkKafkaConsumer , for console-consumer it
>>>> shows them in the Consumers list.
>>>>
>>>> I have set the required parameters for the kafka
>>>> consumer while running the application.
>>>>
>>>> Has anyone faced this issue ?
>>>> I am using Kafka 0.9.0.1
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> If you reply to this email, your message will be added
>>> to the discussion below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>> To start a new topic under Apache Flink User Mailing
>>> List archive., email<a
>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>> target="_top" rel="nofollow" link="external"
>>> class="">[hidden email]
>>> To unsubscribe from Apache Flink User Mailing List
>>> archive.,click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> View this message in context:Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>> Sent from theApache Flink User Mailing List archive. mailing
>>> list archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>> <http://nabble.com/>.
>>
>>
>>
>> ------------------------------------------------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>
>> To start a new topic under Apache Flink User Mailing List
>> archive., email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive.,
>> click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>> Sent from the Apache Flink User Mailing List archive. mailing
>> list archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>
> To start a new topic under Apache Flink User Mailing List
> archive., email [hidden email]
> </user/SendEmail.jtp?type=node&node=9978&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
> Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> at Nabble.com.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Daniel,
I have the same properties set for the consumer and the same code
*brokerspath only needed if you have set it on kafka config* -> I did not
get this, do you mean to check the brokerspath in conf/server.properties
file ? I have even tried by setting offset.storage property to zookeeeper,
but still not getting the consumers listed
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hello,
>
> This is my config.
>
> On kafka props :
>
> val props = new Properties()
>
> props.setProperty("zookeeper.connect", zkHosts)
> props.setProperty("bootstrap.servers", kafHosts)
> props.setProperty("group.id", "prod")
> props.setProperty("auto.offset.reset", "earliest")
>
> Now for zkHosts beware that all your hosts quorum has to be included.
>
> For instances you have zk1 and zk2 and zk3 to form a quorum.
>
> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
> .
>
> brokerspath only needed if you have set it on kafka config. Ignore it
> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>
> After that -> val source = env.addSource(new FlinkKafkaConsumer09[String](KAFKA_TOPIC,
> new SimpleStringSchema(), props))
>
> Then on kafkamanager -> consumers I have the groupID prod.
>
> Hope it helps.
>
> Best Regards,
>
> Daniel Santos
> On 11/08/2016 08:45 AM, vinay patil wrote:
>
> Hi Limbo,
>
> I am using 0.9, I am not able to see updated results even after refreshing.
> There is some property that we have to set in order to make this work
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing List
> archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>
>> I am using kafka 0.8, just refresh the page and you will see the updated
>> results, it’s not auto update.
>>
>> There is our logstash consumer:
>>
>>
>>
>> and this is our flink consumer:
>>
>>
>> I find the flink consumer just write the offset of the kafka partition to
>> zookeeper without owner and ids,
>> so we can’t find the consumer in the manager page, we can only find the
>> offset info.
>>
>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>>
>> Hi Limbo,
>>
>> I can see the lag by using that URL, but the Lag there is not showing
>> updated results, it does not change, also if you try to to change the
>> consumer group value it will still show you the same value instead of
>> saying consumer group does not exist or similar kind of error :)
>>
>> According to documentation of 0.9.x the offsets are stored in Kafka, but
>> we can set offset.storage property to zookeeper.
>> Even by setting this I am not getting the consumer listed.
>>
>> Kafka cli command also does not show this consumer
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List
>> archive.] <<a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I have the same problem, I think the reason is that the consumer of
>>> flink use the low level API,
>>> and when I type the group name in manager url I can get the lag of the
>>> flink consumer, like this:
>>>
>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>> sumer_name>/topic/<topic_name>/type/ZK
>>>
>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>>
>>> Hello,
>>>
>>> I have been using that setup.
>>> From my understanding, if one desires to see the offset being consumed
>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it
>>> will only serve as a view of progress.
>>>
>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>> purposes only.
>>>
>>> Best Regards,
>>> Daniel Santos
>>>
>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>> other Kafka metrics, however I am not able to see the consumers when I use
>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>> list.
>>>>
>>>> I have set the required parameters for the kafka consumer while running
>>>> the application.
>>>>
>>>> Has anyone faced this issue ?
>>>> I am using Kafka 0.9.0.1
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>> To start a new topic under Apache Flink User Mailing List archive., email
>>> <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
>> Nabble.com <http://nabble.com/>.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9965.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9976.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by Daniel Santos <ds...@cryptolab.net>.
Hello,
This is my config.
On kafka props :
val props = new Properties()
props.setProperty("zookeeper.connect", zkHosts)
props.setProperty("bootstrap.servers", kafHosts)
props.setProperty("group.id", "prod")
props.setProperty("auto.offset.reset", "earliest")
Now for zkHosts beware that all your hosts quorum has to be included.
For instances you have zk1 and zk2 and zk3 to form a quorum.
Then it will result in zkHosts being ->
zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
brokerspath only needed if you have set it on kafka config. Ignore it
otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
After that -> val source = env.addSource(new
FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(), props))
Then on kafkamanager -> consumers I have the groupID prod.
Hope it helps.
Best Regards,
Daniel Santos
On 11/08/2016 08:45 AM, vinay patil wrote:
> Hi Limbo,
>
> I am using 0.9, I am not able to see updated results even after
> refreshing.
> There is some property that we have to set in order to make this work
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
> List archive.] <[hidden email]
> </user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>
> I am using kafka 0.8, just refresh the page and you will see the
> updated results, it\u2019s not auto update.
>
> There is our logstash consumer:
>
>
>
> and this is our flink consumer:
>
>
> I find the flink consumer just write the offset of the kafka
> partition to zookeeper without owner and ids,
> so we can\u2019t find the consumer in the manager page, we can only
> find the offset info.
>
>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0b\u534812:02\uff0cvinay patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> \u5199\u9053\uff1a
>>
>> Hi Limbo,
>>
>> I can see the lag by using that URL, but the Lag there is not
>> showing updated results, it does not change, also if you try to
>> to change the consumer group value it will still show you the
>> same value instead of saying consumer group does not exist or
>> similar kind of error :)
>>
>> According to documentation of 0.9.x the offsets are stored in
>> Kafka, but we can set offset.storage property to zookeeper.
>> Even by setting this I am not getting the consumer listed.
>>
>> Kafka cli command also does not show this consumer
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User
>> Mailing List archive.]<<a
>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>> target="_top" rel="nofollow" link="external" class="">[hidden
>> email]>wrote:
>>
>> Hi,
>>
>> I have the same problem, I think the reason is that the
>> consumer of flink use the low level API,
>> and when I type the group name in manager url I can get the
>> lag of the flink consumer, like this:
>>
>> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>> <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>
>>> \u5728 2016\u5e7411\u67088\u65e5\uff0c\u4e0a\u53485:12\uff0cDaniel Santos <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> \u5199\u9053\uff1a
>>>
>>> Hello,
>>>
>>> I have been using that setup.
>>> From my understanding, if one desires to see the offset
>>> being consumed by Flink on KafkaManger, one has to set it up
>>> with zookeeper. On 0.9 it will only serve as a view of progress.
>>>
>>> Basically what's mandatory on 0.8 is optional on 0.9, and
>>> for viewing purposes only.
>>>
>>> Best Regards,
>>> Daniel Santos
>>>
>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil
>>> <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>
>>> Hi,
>>>
>>> I am monitoring Kafka using KafkaManager for checking
>>> offset lag and other Kafka metrics, however I am not
>>> able to see the consumers when I use FlinkKafkaConsumer
>>> , for console-consumer it shows them in the Consumers list.
>>>
>>> I have set the required parameters for the kafka
>>> consumer while running the application.
>>>
>>> Has anyone faced this issue ?
>>> I am using Kafka 0.9.0.1
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>> To start a new topic under Apache Flink User Mailing List
>> archive., email<a
>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>> target="_top" rel="nofollow" link="external" class="">[hidden
>> email]
>> To unsubscribe from Apache Flink User Mailing List
>> archive.,click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>> ------------------------------------------------------------------------
>> View this message in context:Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>> Sent from theApache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>> <http://nabble.com/>.
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>
> To start a new topic under Apache Flink User Mailing List
> archive., email [hidden email]
> </user/SendEmail.jtp?type=node&node=9969&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
> Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> at Nabble.com.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Limbo,
I am using 0.9, I am not able to see updated results even after refreshing.
There is some property that we have to set in order to make this work
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing List
archive.] <ml...@n4.nabble.com> wrote:
> I am using kafka 0.8, just refresh the page and you will see the updated
> results, it’s not auto update.
>
> There is our logstash consumer:
>
>
>
> and this is our flink consumer:
>
>
> I find the flink consumer just write the offset of the kafka partition to
> zookeeper without owner and ids,
> so we can’t find the consumer in the manager page, we can only find the
> offset info.
>
> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>
> Hi Limbo,
>
> I can see the lag by using that URL, but the Lag there is not showing
> updated results, it does not change, also if you try to to change the
> consumer group value it will still show you the same value instead of
> saying consumer group does not exist or similar kind of error :)
>
> According to documentation of 0.9.x the offsets are stored in Kafka, but
> we can set offset.storage property to zookeeper.
> Even by setting this I am not getting the consumer listed.
>
> Kafka cli command also does not show this consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List
> archive.] <<a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
> target="_top" rel="nofollow" link="external" class="">[hidden email]>
> wrote:
>
>> Hi,
>>
>> I have the same problem, I think the reason is that the consumer of flink
>> use the low level API,
>> and when I type the group name in manager url I can get the lag of the
>> flink consumer, like this:
>>
>> http://your_manager_url/clusters/<kafka_name>/consumers/<
>> consumer_name>/topic/<topic_name>/type/ZK
>>
>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>
>> Hello,
>>
>> I have been using that setup.
>> From my understanding, if one desires to see the offset being consumed by
>> Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will
>> only serve as a view of progress.
>>
>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>> purposes only.
>>
>> Best Regards,
>> Daniel Santos
>>
>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>
>>> Hi,
>>>
>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>> other Kafka metrics, however I am not able to see the consumers when I use
>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>> list.
>>>
>>> I have set the required parameters for the kafka consumer while running
>>> the application.
>>>
>>> Has anyone faced this issue ?
>>> I am using Kafka 0.9.0.1
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9962.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com <http://nabble.com/>.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9965.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by limbo <46...@qq.com>.
I am using kafka 0.8, just refresh the page and you will see the updated results, it’s not auto update.
There is our logstash consumer:
and this is our flink consumer:
I find the flink consumer just write the offset of the kafka partition to zookeeper without owner and ids,
so we can’t find the consumer in the manager page, we can only find the offset info.
> 在 2016年11月8日，下午12:02，vinay patil <vi...@gmail.com> 写道：
>
> Hi Limbo,
>
> I can see the lag by using that URL, but the Lag there is not showing updated results, it does not change, also if you try to to change the consumer group value it will still show you the same value instead of saying consumer group does not exist or similar kind of error :)
>
> According to documentation of 0.9.x the offsets are stored in Kafka, but we can set offset.storage property to zookeeper.
> Even by setting this I am not getting the consumer listed.
>
> Kafka cli command also does not show this consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List archive.] <[hidden email] <x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0>> wrote:
> Hi,
>
> I have the same problem, I think the reason is that the consumer of flink use the low level API,
> and when I type the group name in manager url I can get the lag of the flink consumer, like this:
>
> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>
>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email] <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>
>> Hello,
>>
>> I have been using that setup.
>> From my understanding, if one desires to see the offset being consumed by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will only serve as a view of progress.
>>
>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing purposes only.
>>
>> Best Regards,
>> Daniel Santos
>>
>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>> Hi,
>>
>> I am monitoring Kafka using KafkaManager for checking offset lag and other Kafka metrics, however I am not able to see the consumers when I use FlinkKafkaConsumer , for console-consumer it shows them in the Consumers list.
>>
>> I have set the required parameters for the kafka consumer while running the application.
>>
>> Has anyone faced this issue ?
>> I am using Kafka 0.9.0.1
>>
>> Regards,
>> Vinay Patil
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://911FFAC4-9E12-48DB-AC3D-1BDD5A374626>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Kafka Monitoring <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com <http://nabble.com/>.
Re: Kafka Monitoring
Posted by vinay patil <vi...@gmail.com>.
Hi Limbo,
I can see the lag by using that URL, but the Lag there is not showing
updated results, it does not change, also if you try to to change the
consumer group value it will still show you the same value instead of
saying consumer group does not exist or similar kind of error :)
According to documentation of 0.9.x the offsets are stored in Kafka, but we
can set offset.storage property to zookeeper.
Even by setting this I am not getting the consumer listed.
Kafka cli command also does not show this consumer
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List
archive.] <ml...@n4.nabble.com> wrote:
> Hi,
>
> I have the same problem, I think the reason is that the consumer of flink
> use the low level API,
> and when I type the group name in manager url I can get the lag of the
> flink consumer, like this:
>
> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/
> topic/<topic_name>/type/ZK
>
> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>
> Hello,
>
> I have been using that setup.
> From my understanding, if one desires to see the offset being consumed by
> Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will
> only serve as a view of progress.
>
> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
> purposes only.
>
> Best Regards,
> Daniel Santos
>
> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>
>> Hi,
>>
>> I am monitoring Kafka using KafkaManager for checking offset lag and
>> other Kafka metrics, however I am not able to see the consumers when I use
>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>> list.
>>
>> I have set the required parameters for the kafka consumer while running
>> the application.
>>
>> Has anyone faced this issue ?
>> I am using Kafka 0.9.0.1
>>
>> Regards,
>> Vinay Patil
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9962.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Kafka Monitoring
Posted by limbo <46...@qq.com>.
Hi,
I have the same problem, I think the reason is that the consumer of flink use the low level API,
and when I type the group name in manager url I can get the lag of the flink consumer, like this:
http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
> 在 2016年11月8日，上午5:12，Daniel Santos <ds...@cryptolab.net> 写道：
>
> Hello,
>
> I have been using that setup.
> From my understanding, if one desires to see the offset being consumed by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will only serve as a view of progress.
>
> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing purposes only.
>
> Best Regards,
> Daniel Santos
>
> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <vi...@gmail.com> wrote:
> Hi,
>
> I am monitoring Kafka using KafkaManager for checking offset lag and other Kafka metrics, however I am not able to see the consumers when I use FlinkKafkaConsumer , for console-consumer it shows them in the Consumers list.
>
> I have set the required parameters for the kafka consumer while running the application.
>
> Has anyone faced this issue ?
> I am using Kafka 0.9.0.1
>
> Regards,
> Vinay Patil
Re: Kafka Monitoring
Posted by Daniel Santos <ds...@cryptolab.net>.
Hello,
I have been using that setup.
From my understanding, if one desires to see the offset being consumed by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it will only serve as a view of progress.
Basically what's mandatory on 0.8 is optional on 0.9, and for viewing purposes only.
Best Regards,
Daniel Santos
On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <vi...@gmail.com> wrote:
>Hi,
>
>I am monitoring Kafka using KafkaManager for checking offset lag and
>other
>Kafka metrics, however I am not able to see the consumers when I use
>FlinkKafkaConsumer , for console-consumer it shows them in the
>Consumers
>list.
>
>I have set the required parameters for the kafka consumer while running
>the
>application.
>
>Has anyone faced this issue ?
>I am using Kafka 0.9.0.1
>
>Regards,
>Vinay Patil