You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gary Taylor <ga...@gmail.com> on 2018/03/25 07:46:50 UTC
Developing with kafka and other non jvm languages
Hi,
This is a fairly generic question but has some specifics too
Ill ask the specific first - I am trying to use golang to talk to kafka and it works, but a fairly important part of my application is to subscribe to information in many topics where the topic is matched server side and will include new topics added since the subscribe. I have read that kafka can do this, but I cannot see any way of asking it to do it via golang apart from with the library https://github.com/confluentinc/confluent-kafka-go <https://github.com/confluentinc/confluent-kafka-go> - which uses a library called librdkafka which I would have to compile and I am struggling as I am using an arm processor (long story behind that one - but I don’t think this solution will ever work with an arm processor). All other libraries seem to not offer this. Is this because of some underlying restriction in kafka or do you think it is going to be a case of keep hunting until I find one that does ?
Then the more generic question - I get the feeling that Scala and Java are first class citizens when it comes to using kafka. Whilst I love Scala - I am learning golang and am more generally a ruby developer. Am I likely go get a ‘second class’ service with these other languages - with restrictions around zookeeper for example (I think I read something about having to have knowledge of which partition to subscribe to - where zookeeper would normally track this but there is no API for it) ?
Many Thanks
Gary Taylor
Re: Developing with kafka and other non jvm languages
Posted by Gary Taylor <ga...@hismessages.com>.
Hi Matthias,
Thanks for the reply. Ill start hunting around to see what I can find with regards to clients. I had just looked at the 2 most popular ones but you never know, the features I want may be in another library
Cheers
Gary
> On 25 Mar 2018, at 20:24, Matthias J. Sax <ma...@confluent.io> wrote:
>
> Gary,
>
> The Apache Kafka project itself, only maintains Java clients -- thus,
> your are right that those are the primary and best supported clients.
> (Kafka used to have Scala clients, but those are all deprecated now and
> will be removed eventually.)
>
> Clients in other languages are not part of Apache Kafka project itself
> but developed by third parties. Thus, the quality and feature set of
> those clients can vary. A list of known third party clients is
> maintained in the Kafka Wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients
>
> If a client does not support a feature, it's a client limitation only.
> The Kafka protocol is a binary protocol and thus it's up to the client
> to (fully) implement the protocol -- the broker does not even know in
> which language a client is written. (cf
> https://kafka.apache.org/protocol) Thus, you should open feature
> requests for clients to support pattern subscription if it's missing and
> you need it.
>
> If you need help to get librdkafka to compile on arm, you might want to
> reach out to on Github (https://github.com/edenhill/librdkafka) or
> Confluent mailing list
> (https://groups.google.com/forum/#!forum/confluent-platform) or
> Confluent Slack (https://launchpass.com/confluentcommunity).
>
> Hope this helps!
>
>
> -Matthias
>
>
>
>
> On 3/25/18 12:46 AM, Gary Taylor wrote:
>> Hi,
>> This is a fairly generic question but has some specifics too
>>
>> Ill ask the specific first - I am trying to use golang to talk to kafka and it works, but a fairly important part of my application is to subscribe to information in many topics where the topic is matched server side and will include new topics added since the subscribe. I have read that kafka can do this, but I cannot see any way of asking it to do it via golang apart from with the library https://github.com/confluentinc/confluent-kafka-go <https://github.com/confluentinc/confluent-kafka-go> - which uses a library called librdkafka which I would have to compile and I am struggling as I am using an arm processor (long story behind that one - but I don’t think this solution will ever work with an arm processor). All other libraries seem to not offer this. Is this because of some underlying restriction in kafka or do you think it is going to be a case of keep hunting until I find one that does ?
>>
>> Then the more generic question - I get the feeling that Scala and Java are first class citizens when it comes to using kafka. Whilst I love Scala - I am learning golang and am more generally a ruby developer. Am I likely go get a ‘second class’ service with these other languages - with restrictions around zookeeper for example (I think I read something about having to have knowledge of which partition to subscribe to - where zookeeper would normally track this but there is no API for it) ?
>>
>> Many Thanks
>>
>> Gary Taylor
>>
>
Re: Developing with kafka and other non jvm languages
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Gary,
The Apache Kafka project itself, only maintains Java clients -- thus,
your are right that those are the primary and best supported clients.
(Kafka used to have Scala clients, but those are all deprecated now and
will be removed eventually.)
Clients in other languages are not part of Apache Kafka project itself
but developed by third parties. Thus, the quality and feature set of
those clients can vary. A list of known third party clients is
maintained in the Kafka Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Clients
If a client does not support a feature, it's a client limitation only.
The Kafka protocol is a binary protocol and thus it's up to the client
to (fully) implement the protocol -- the broker does not even know in
which language a client is written. (cf
https://kafka.apache.org/protocol) Thus, you should open feature
requests for clients to support pattern subscription if it's missing and
you need it.
If you need help to get librdkafka to compile on arm, you might want to
reach out to on Github (https://github.com/edenhill/librdkafka) or
Confluent mailing list
(https://groups.google.com/forum/#!forum/confluent-platform) or
Confluent Slack (https://launchpass.com/confluentcommunity).
Hope this helps!
-Matthias
On 3/25/18 12:46 AM, Gary Taylor wrote:
> Hi,
> This is a fairly generic question but has some specifics too
>
> Ill ask the specific first - I am trying to use golang to talk to kafka and it works, but a fairly important part of my application is to subscribe to information in many topics where the topic is matched server side and will include new topics added since the subscribe. I have read that kafka can do this, but I cannot see any way of asking it to do it via golang apart from with the library https://github.com/confluentinc/confluent-kafka-go <https://github.com/confluentinc/confluent-kafka-go> - which uses a library called librdkafka which I would have to compile and I am struggling as I am using an arm processor (long story behind that one - but I don’t think this solution will ever work with an arm processor). All other libraries seem to not offer this. Is this because of some underlying restriction in kafka or do you think it is going to be a case of keep hunting until I find one that does ?
>
> Then the more generic question - I get the feeling that Scala and Java are first class citizens when it comes to using kafka. Whilst I love Scala - I am learning golang and am more generally a ruby developer. Am I likely go get a ‘second class’ service with these other languages - with restrictions around zookeeper for example (I think I read something about having to have knowledge of which partition to subscribe to - where zookeeper would normally track this but there is no API for it) ?
>
> Many Thanks
>
> Gary Taylor
>
Re: Developing with kafka and other non jvm languages
Posted by Gary Taylor <ga...@hismessages.com>.
Hi Robin,
Maybe this is where I have been getting confused - when I read the docs about the regex support for topics - I probably made a big assumption it was a server side thing. Is this not the case ? would explain why Ive not found much support for it lol
Cheers
Gary
> On 26 Mar 2018, at 16:18, Dmitriy Vsekhvalnov <dv...@gmail.com> wrote:
>
> Hi Gary,
>
> don't have experience with other go libs (they seems to be way younger),
> but Sarama is quite low level, which is both at same time powerful and to
> some extent more complicated to work with.
>
> With pure Sarama client you have to implement wildcard (or pattern based)
> topic subscription yourself. In all clients in all languages it is usually
> as simple as:
>
> - every X seconds (minutes, e.tc.) refresh metadata (see
> Client.RefreshMetadata(..))
> - diff new metadata with old metadata = removed topics/added topics
> - stop consuming for removed topics
> - start consuming for new topics
>
>
>
> On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor <ga...@hismessages.com>
> wrote:
>
>> Hi Robin - thanks for that. I started off with sarama, but found it did
>> not (seem to) support regex matching of the topics - I tried all sorts of
>> ‘regex as strings’ as the topic - but it has to be a string according to
>> the interface.
>>
>> I will dig deeper though and look at those other clients too
>>
>> Cheers
>>
>> Gary
>>
>>> On 26 Mar 2018, at 09:54, Robin Bartholdson <
>> robin.bartholdson@formulate.se> wrote:
>>>
>>> For go I would seriously consider using a pure go client, for the
>> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <
>> https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
>>>
>>> There is now a wealth of pure go Kafka clients to choose from:
>>>
>>> - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama>
>>> - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/
>> kafka-go/>
>>> - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka>
>>>
>>> The functionality you are looking for is built into sarama-cluster, the
>> package that provides consumer group functionality on top of sarama:
>>>
>>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <
>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
>>>
>>> If you for some reason don’t want to use sarama-cluster, you can
>> implement it yourself quite easily: Every now and then, as metadata is
>> being refreshed, just cycle through the list of topics and determine if any
>> of the topics are worth subscribing to:
>>>
>>> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448
>> <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448>
>>>
>>> For each of the clients these are the relevant methods to use to grab
>> the list of topics:
>>>
>>> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <
>> https://godoc.org/github.com/Shopify/sarama#Client>
>>> - kafka-go: ReadPartitions(): https://godoc.org/github.com/
>> segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/
>> segmentio/kafka-go#Conn.ReadPartitions>
>>> - optiopay/kafka: Metadata(): https://godoc.org/github.com/
>> optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/
>> optiopay/kafka#Broker.Metadata>
>>>
>>> -Robin
>>>
>>>
>>>> On 25 Mar 2018, at 09:46, Gary Taylor <ga...@gmail.com>
>> wrote:
>>>>
>>>> Hi,
>>>> This is a fairly generic question but has some specifics too
>>>>
>>>> Ill ask the specific first - I am trying to use golang to talk to kafka
>> and it works, but a fairly important part of my application is to subscribe
>> to information in many topics where the topic is matched server side and
>> will include new topics added since the subscribe. I have read that kafka
>> can do this, but I cannot see any way of asking it to do it via golang
>> apart from with the library https://github.com/
>> confluentinc/confluent-kafka-go <https://github.com/
>> confluentinc/confluent-kafka-go> - which uses a library called librdkafka
>> which I would have to compile and I am struggling as I am using an arm
>> processor (long story behind that one - but I don’t think this solution
>> will ever work with an arm processor). All other libraries seem to not
>> offer this. Is this because of some underlying restriction in kafka or do
>> you think it is going to be a case of keep hunting until I find one that
>> does ?
>>>>
>>>> Then the more generic question - I get the feeling that Scala and Java
>> are first class citizens when it comes to using kafka. Whilst I love Scala
>> - I am learning golang and am more generally a ruby developer. Am I likely
>> go get a ‘second class’ service with these other languages - with
>> restrictions around zookeeper for example (I think I read something about
>> having to have knowledge of which partition to subscribe to - where
>> zookeeper would normally track this but there is no API for it) ?
>>>>
>>>> Many Thanks
>>>>
>>>> Gary Taylor
>>>
>>
>>
Re: Developing with kafka and other non jvm languages
Posted by Dmitriy Vsekhvalnov <dv...@gmail.com>.
Hi Gary,
don't have experience with other go libs (they seems to be way younger),
but Sarama is quite low level, which is both at same time powerful and to
some extent more complicated to work with.
With pure Sarama client you have to implement wildcard (or pattern based)
topic subscription yourself. In all clients in all languages it is usually
as simple as:
- every X seconds (minutes, e.tc.) refresh metadata (see
Client.RefreshMetadata(..))
- diff new metadata with old metadata = removed topics/added topics
- stop consuming for removed topics
- start consuming for new topics
On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor <ga...@hismessages.com>
wrote:
> Hi Robin - thanks for that. I started off with sarama, but found it did
> not (seem to) support regex matching of the topics - I tried all sorts of
> ‘regex as strings’ as the topic - but it has to be a string according to
> the interface.
>
> I will dig deeper though and look at those other clients too
>
> Cheers
>
> Gary
>
> > On 26 Mar 2018, at 09:54, Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
> >
> > For go I would seriously consider using a pure go client, for the
> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <
> https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
> >
> > There is now a wealth of pure go Kafka clients to choose from:
> >
> > - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama>
> > - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/
> kafka-go/>
> > - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka>
> >
> > The functionality you are looking for is built into sarama-cluster, the
> package that provides consumer group functionality on top of sarama:
> >
> > https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <
> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
> >
> > If you for some reason don’t want to use sarama-cluster, you can
> implement it yourself quite easily: Every now and then, as metadata is
> being refreshed, just cycle through the list of topics and determine if any
> of the topics are worth subscribing to:
> >
> > https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448
> <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448>
> >
> > For each of the clients these are the relevant methods to use to grab
> the list of topics:
> >
> > - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <
> https://godoc.org/github.com/Shopify/sarama#Client>
> > - kafka-go: ReadPartitions(): https://godoc.org/github.com/
> segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/
> segmentio/kafka-go#Conn.ReadPartitions>
> > - optiopay/kafka: Metadata(): https://godoc.org/github.com/
> optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/
> optiopay/kafka#Broker.Metadata>
> >
> > -Robin
> >
> >
> >> On 25 Mar 2018, at 09:46, Gary Taylor <ga...@gmail.com>
> wrote:
> >>
> >> Hi,
> >> This is a fairly generic question but has some specifics too
> >>
> >> Ill ask the specific first - I am trying to use golang to talk to kafka
> and it works, but a fairly important part of my application is to subscribe
> to information in many topics where the topic is matched server side and
> will include new topics added since the subscribe. I have read that kafka
> can do this, but I cannot see any way of asking it to do it via golang
> apart from with the library https://github.com/
> confluentinc/confluent-kafka-go <https://github.com/
> confluentinc/confluent-kafka-go> - which uses a library called librdkafka
> which I would have to compile and I am struggling as I am using an arm
> processor (long story behind that one - but I don’t think this solution
> will ever work with an arm processor). All other libraries seem to not
> offer this. Is this because of some underlying restriction in kafka or do
> you think it is going to be a case of keep hunting until I find one that
> does ?
> >>
> >> Then the more generic question - I get the feeling that Scala and Java
> are first class citizens when it comes to using kafka. Whilst I love Scala
> - I am learning golang and am more generally a ruby developer. Am I likely
> go get a ‘second class’ service with these other languages - with
> restrictions around zookeeper for example (I think I read something about
> having to have knowledge of which partition to subscribe to - where
> zookeeper would normally track this but there is no API for it) ?
> >>
> >> Many Thanks
> >>
> >> Gary Taylor
> >
>
>
Re: Developing with kafka and other non jvm languages
Posted by Gary Taylor <ga...@hismessages.com>.
Hi Robin - thanks for that. I started off with sarama, but found it did not (seem to) support regex matching of the topics - I tried all sorts of ‘regex as strings’ as the topic - but it has to be a string according to the interface.
I will dig deeper though and look at those other clients too
Cheers
Gary
> On 26 Mar 2018, at 09:54, Robin Bartholdson <ro...@formulate.se> wrote:
>
> For go I would seriously consider using a pure go client, for the reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
>
> There is now a wealth of pure go Kafka clients to choose from:
>
> - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama>
> - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/kafka-go/>
> - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka>
>
> The functionality you are looking for is built into sarama-cluster, the package that provides consumer group functionality on top of sarama:
>
> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
>
> If you for some reason don’t want to use sarama-cluster, you can implement it yourself quite easily: Every now and then, as metadata is being refreshed, just cycle through the list of topics and determine if any of the topics are worth subscribing to:
>
> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448>
>
> For each of the clients these are the relevant methods to use to grab the list of topics:
>
> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <https://godoc.org/github.com/Shopify/sarama#Client>
> - kafka-go: ReadPartitions(): https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions>
> - optiopay/kafka: Metadata(): https://godoc.org/github.com/optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/optiopay/kafka#Broker.Metadata>
>
> -Robin
>
>
>> On 25 Mar 2018, at 09:46, Gary Taylor <ga...@gmail.com> wrote:
>>
>> Hi,
>> This is a fairly generic question but has some specifics too
>>
>> Ill ask the specific first - I am trying to use golang to talk to kafka and it works, but a fairly important part of my application is to subscribe to information in many topics where the topic is matched server side and will include new topics added since the subscribe. I have read that kafka can do this, but I cannot see any way of asking it to do it via golang apart from with the library https://github.com/confluentinc/confluent-kafka-go <https://github.com/confluentinc/confluent-kafka-go> - which uses a library called librdkafka which I would have to compile and I am struggling as I am using an arm processor (long story behind that one - but I don’t think this solution will ever work with an arm processor). All other libraries seem to not offer this. Is this because of some underlying restriction in kafka or do you think it is going to be a case of keep hunting until I find one that does ?
>>
>> Then the more generic question - I get the feeling that Scala and Java are first class citizens when it comes to using kafka. Whilst I love Scala - I am learning golang and am more generally a ruby developer. Am I likely go get a ‘second class’ service with these other languages - with restrictions around zookeeper for example (I think I read something about having to have knowledge of which partition to subscribe to - where zookeeper would normally track this but there is no API for it) ?
>>
>> Many Thanks
>>
>> Gary Taylor
>
Re: Developing with kafka and other non jvm languages
Posted by Robin Bartholdson <ro...@formulate.se>.
For go I would seriously consider using a pure go client, for the reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
There is now a wealth of pure go Kafka clients to choose from:
- https://github.com/Shopify/sarama <https://github.com/Shopify/sarama>
- https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/kafka-go/>
- https://github.com/optiopay/kafka <https://github.com/optiopay/kafka>
The functionality you are looking for is built into sarama-cluster, the package that provides consumer group functionality on top of sarama:
https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
If you for some reason don’t want to use sarama-cluster, you can implement it yourself quite easily: Every now and then, as metadata is being refreshed, just cycle through the list of topics and determine if any of the topics are worth subscribing to:
https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448>
For each of the clients these are the relevant methods to use to grab the list of topics:
- sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <https://godoc.org/github.com/Shopify/sarama#Client>
- kafka-go: ReadPartitions(): https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions>
- optiopay/kafka: Metadata(): https://godoc.org/github.com/optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/optiopay/kafka#Broker.Metadata>
-Robin
> On 25 Mar 2018, at 09:46, Gary Taylor <ga...@gmail.com> wrote:
>
> Hi,
> This is a fairly generic question but has some specifics too
>
> Ill ask the specific first - I am trying to use golang to talk to kafka and it works, but a fairly important part of my application is to subscribe to information in many topics where the topic is matched server side and will include new topics added since the subscribe. I have read that kafka can do this, but I cannot see any way of asking it to do it via golang apart from with the library https://github.com/confluentinc/confluent-kafka-go <https://github.com/confluentinc/confluent-kafka-go> - which uses a library called librdkafka which I would have to compile and I am struggling as I am using an arm processor (long story behind that one - but I don’t think this solution will ever work with an arm processor). All other libraries seem to not offer this. Is this because of some underlying restriction in kafka or do you think it is going to be a case of keep hunting until I find one that does ?
>
> Then the more generic question - I get the feeling that Scala and Java are first class citizens when it comes to using kafka. Whilst I love Scala - I am learning golang and am more generally a ruby developer. Am I likely go get a ‘second class’ service with these other languages - with restrictions around zookeeper for example (I think I read something about having to have knowledge of which partition to subscribe to - where zookeeper would normally track this but there is no API for it) ?
>
> Many Thanks
>
> Gary Taylor