Questions About Kafka Source

[舒琦 <sh...@eefung.com>]

Hi,


I constructed a DAG as show blowing, “kafka source”consumes messages from kafka topic “webs”, its metrics shows that it consumes lots of messages, but actually there is no messages handled and I also can’t find active group under topic “webs”, the log is ok too.


I just wonder the properties of kafka for zks and brokers, if there is a list of zookeeper servers, should I use comma to separate? just like below:


zks=zk1:3181,zk2:3181,zk3:3181
brokers=kfk1:9096,kfk2:9096,kfk3:9096


Thanks for your help.






————————
Qi Shu

Re: Questions About Kafka Source

[Manu Zhang <ow...@gmail.com>]

Hi Qi,

Neither the red ballon nor the message receive message throughput means any
message has been consumed by KafkaSource. Those are messages source send to
itself to trigger next Task execution. The metrics is a bit confusing and I
think we need to fix this.

Yes, both zookeeper servers and kafka brokers configs are comma-separated
list strings. One way to check whether your configurations is correct it to
consume from the topic using kafka-console-consumer. This also makes sure
the topic has data to consume.

Hope this helps.

Thanks,
Manu

On Tue, May 3, 2016 at 12:07 PM 舒琦 <sh...@eefung.com> wrote:

> Hi,
>
>
> I constructed a DAG as show blowing, “kafka source”consumes messages from
> kafka topic “webs”, its metrics shows  that it consumes lots of messages,
> but actually there is no messages handled and I also can’t find active
> group under topic “webs”, the log is ok too.
>
>
> I just wonder the properties of kafka for zks and brokers, if there is a
> list of zookeeper servers, should I use comma to separate? just like below:
>
>
> zks=zk1:3181,zk2:3181,zk3:3181
>
> brokers=kfk1:9096,kfk2:9096,kfk3:9096
>
>
> Thanks for your help.
>
>
>
>
> ————————
> Qi Shu
>