You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Krishna Raj <re...@gmail.com> on 2014/11/25 00:57:37 UTC
Does Kafka Producer service ?
Hello Amazing Kafka Creators & User,
I have learnt and use kafka in our Production system, so you can count my
understanding as intermediate.
With the statement that "Kafka has solved the Scalability and Availability
needs for a large scale message publish/subscribe system", I understand
that having a Producer Service which sits in between the Application and
the Producer defects the one major purpose of Kafka.
So, my question is, How to loosely couple Kafka with my Production
Application ?
The reason being, I wish to do all producer code and Kafka library
maintenance without affecting my large scale Production system. Its not an
easy thing to buy a window to these type of changes done on a large scale
production application :)
Any advice on how this can be achieved(even moderately) will greatly help ?
Thanks,
Krishna Raj
Re: Does Kafka Producer service ?
Posted by Harsha <ka...@harsha.io>.
If you want to separate your main application from producer logic you
can use or write something similar to
https://github.com/mozilla-metrics/bagheera . Basically have a service
that provides an REST api which your main application can call and has a
Kafkaproducer to write to kafka.
-Harsha
On Tue, Nov 25, 2014, at 06:47 PM, Jun Rao wrote:
> Assuming that you use a single producer in the async mode, the Kafka
> overhead should be limited to a single thread. Using a cheaper
> compression
> codec such as snappy will also help reducing the CPU load.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj
> <re...@gmail.com>
> wrote:
>
> > Hi Jun,
> >
> > Thanks for replying back on this. Appreciated.
> >
> > I do understand that the Kafka Client just needs a protocol compatibility
> > with the Application which is producing the messages.
> >
> > To clarity a bit more:
> >
> > I witnessed a scenario where a large scale website uses the Kafka Library
> > in their Web Application. So in this case, the Kafka libraries are tied to
> > the Application which are served by Web Servers.
> >
> > So, When there was an issue caused by Kafka related to CPU usage, the team
> > wanted to do a patch. In this case, in order to do a patch, they had to
> > create a new WAR package and deploy again in Web Server which is a
> > significant effort.
> >
> > I totally understand that having a layer like Logging service in between
> > Kafka and the Application will totally defect the purpose for Kafka.
> >
> > And I would love to know your advice how best to handle these type of
> > maintenance.
> >
> > Thanks,
> > Krishna Raj
> >
> >
> >
> >
> >
> > On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> Could you be a bit more specific about the issue? As long as there is
> >> protocol compatibility btw the Kafka client and the broker, upgrading the
> >> Kafka client library should be easy, right?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
> >> wrote:
> >>
> >>> Hello Amazing Kafka Creators & User,
> >>>
> >>> I have learnt and use kafka in our Production system, so you can count
> >>> my understanding as intermediate.
> >>>
> >>> With the statement that "Kafka has solved the Scalability and
> >>> Availability needs for a large scale message publish/subscribe system", I
> >>> understand that having a Producer Service which sits in between the
> >>> Application and the Producer defects the one major purpose of Kafka.
> >>>
> >>> So, my question is, How to loosely couple Kafka with my Production
> >>> Application ?
> >>>
> >>> The reason being, I wish to do all producer code and Kafka library
> >>> maintenance without affecting my large scale Production system. Its not an
> >>> easy thing to buy a window to these type of changes done on a large scale
> >>> production application :)
> >>>
> >>> Any advice on how this can be achieved(even moderately) will greatly
> >>> help ?
> >>>
> >>> Thanks,
> >>> Krishna Raj
> >>>
> >>
> >>
> >
Re: Does Kafka Producer service ?
Posted by Jun Rao <ju...@gmail.com>.
Assuming that you use a single producer in the async mode, the Kafka
overhead should be limited to a single thread. Using a cheaper compression
codec such as snappy will also help reducing the CPU load.
Thanks,
Jun
On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj <re...@gmail.com>
wrote:
> Hi Jun,
>
> Thanks for replying back on this. Appreciated.
>
> I do understand that the Kafka Client just needs a protocol compatibility
> with the Application which is producing the messages.
>
> To clarity a bit more:
>
> I witnessed a scenario where a large scale website uses the Kafka Library
> in their Web Application. So in this case, the Kafka libraries are tied to
> the Application which are served by Web Servers.
>
> So, When there was an issue caused by Kafka related to CPU usage, the team
> wanted to do a patch. In this case, in order to do a patch, they had to
> create a new WAR package and deploy again in Web Server which is a
> significant effort.
>
> I totally understand that having a layer like Logging service in between
> Kafka and the Application will totally defect the purpose for Kafka.
>
> And I would love to know your advice how best to handle these type of
> maintenance.
>
> Thanks,
> Krishna Raj
>
>
>
>
>
> On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Could you be a bit more specific about the issue? As long as there is
>> protocol compatibility btw the Kafka client and the broker, upgrading the
>> Kafka client library should be easy, right?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
>> wrote:
>>
>>> Hello Amazing Kafka Creators & User,
>>>
>>> I have learnt and use kafka in our Production system, so you can count
>>> my understanding as intermediate.
>>>
>>> With the statement that "Kafka has solved the Scalability and
>>> Availability needs for a large scale message publish/subscribe system", I
>>> understand that having a Producer Service which sits in between the
>>> Application and the Producer defects the one major purpose of Kafka.
>>>
>>> So, my question is, How to loosely couple Kafka with my Production
>>> Application ?
>>>
>>> The reason being, I wish to do all producer code and Kafka library
>>> maintenance without affecting my large scale Production system. Its not an
>>> easy thing to buy a window to these type of changes done on a large scale
>>> production application :)
>>>
>>> Any advice on how this can be achieved(even moderately) will greatly
>>> help ?
>>>
>>> Thanks,
>>> Krishna Raj
>>>
>>
>>
>
Re: Does Kafka Producer service ?
Posted by Jun Rao <ju...@gmail.com>.
Assuming that you use a single producer in the async mode, the Kafka
overhead should be limited to a single thread. Using a cheaper compression
codec such as snappy will also help reducing the CPU load.
Thanks,
Jun
On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj <re...@gmail.com>
wrote:
> Hi Jun,
>
> Thanks for replying back on this. Appreciated.
>
> I do understand that the Kafka Client just needs a protocol compatibility
> with the Application which is producing the messages.
>
> To clarity a bit more:
>
> I witnessed a scenario where a large scale website uses the Kafka Library
> in their Web Application. So in this case, the Kafka libraries are tied to
> the Application which are served by Web Servers.
>
> So, When there was an issue caused by Kafka related to CPU usage, the team
> wanted to do a patch. In this case, in order to do a patch, they had to
> create a new WAR package and deploy again in Web Server which is a
> significant effort.
>
> I totally understand that having a layer like Logging service in between
> Kafka and the Application will totally defect the purpose for Kafka.
>
> And I would love to know your advice how best to handle these type of
> maintenance.
>
> Thanks,
> Krishna Raj
>
>
>
>
>
> On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Could you be a bit more specific about the issue? As long as there is
>> protocol compatibility btw the Kafka client and the broker, upgrading the
>> Kafka client library should be easy, right?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
>> wrote:
>>
>>> Hello Amazing Kafka Creators & User,
>>>
>>> I have learnt and use kafka in our Production system, so you can count
>>> my understanding as intermediate.
>>>
>>> With the statement that "Kafka has solved the Scalability and
>>> Availability needs for a large scale message publish/subscribe system", I
>>> understand that having a Producer Service which sits in between the
>>> Application and the Producer defects the one major purpose of Kafka.
>>>
>>> So, my question is, How to loosely couple Kafka with my Production
>>> Application ?
>>>
>>> The reason being, I wish to do all producer code and Kafka library
>>> maintenance without affecting my large scale Production system. Its not an
>>> easy thing to buy a window to these type of changes done on a large scale
>>> production application :)
>>>
>>> Any advice on how this can be achieved(even moderately) will greatly
>>> help ?
>>>
>>> Thanks,
>>> Krishna Raj
>>>
>>
>>
>
Re: Does Kafka Producer service ?
Posted by Krishna Raj <re...@gmail.com>.
Hi Jun,
Thanks for replying back on this. Appreciated.
I do understand that the Kafka Client just needs a protocol compatibility
with the Application which is producing the messages.
To clarity a bit more:
I witnessed a scenario where a large scale website uses the Kafka Library
in their Web Application. So in this case, the Kafka libraries are tied to
the Application which are served by Web Servers.
So, When there was an issue caused by Kafka related to CPU usage, the team
wanted to do a patch. In this case, in order to do a patch, they had to
create a new WAR package and deploy again in Web Server which is a
significant effort.
I totally understand that having a layer like Logging service in between
Kafka and the Application will totally defect the purpose for Kafka.
And I would love to know your advice how best to handle these type of
maintenance.
Thanks,
Krishna Raj
On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <ju...@gmail.com> wrote:
> Could you be a bit more specific about the issue? As long as there is
> protocol compatibility btw the Kafka client and the broker, upgrading the
> Kafka client library should be easy, right?
>
> Thanks,
>
> Jun
>
> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
> wrote:
>
>> Hello Amazing Kafka Creators & User,
>>
>> I have learnt and use kafka in our Production system, so you can count my
>> understanding as intermediate.
>>
>> With the statement that "Kafka has solved the Scalability and
>> Availability needs for a large scale message publish/subscribe system", I
>> understand that having a Producer Service which sits in between the
>> Application and the Producer defects the one major purpose of Kafka.
>>
>> So, my question is, How to loosely couple Kafka with my Production
>> Application ?
>>
>> The reason being, I wish to do all producer code and Kafka library
>> maintenance without affecting my large scale Production system. Its not an
>> easy thing to buy a window to these type of changes done on a large scale
>> production application :)
>>
>> Any advice on how this can be achieved(even moderately) will greatly help
>> ?
>>
>> Thanks,
>> Krishna Raj
>>
>
>
Re: Does Kafka Producer service ?
Posted by Krishna Raj <re...@gmail.com>.
Hi Jun,
Thanks for replying back on this. Appreciated.
I do understand that the Kafka Client just needs a protocol compatibility
with the Application which is producing the messages.
To clarity a bit more:
I witnessed a scenario where a large scale website uses the Kafka Library
in their Web Application. So in this case, the Kafka libraries are tied to
the Application which are served by Web Servers.
So, When there was an issue caused by Kafka related to CPU usage, the team
wanted to do a patch. In this case, in order to do a patch, they had to
create a new WAR package and deploy again in Web Server which is a
significant effort.
I totally understand that having a layer like Logging service in between
Kafka and the Application will totally defect the purpose for Kafka.
And I would love to know your advice how best to handle these type of
maintenance.
Thanks,
Krishna Raj
On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <ju...@gmail.com> wrote:
> Could you be a bit more specific about the issue? As long as there is
> protocol compatibility btw the Kafka client and the broker, upgrading the
> Kafka client library should be easy, right?
>
> Thanks,
>
> Jun
>
> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
> wrote:
>
>> Hello Amazing Kafka Creators & User,
>>
>> I have learnt and use kafka in our Production system, so you can count my
>> understanding as intermediate.
>>
>> With the statement that "Kafka has solved the Scalability and
>> Availability needs for a large scale message publish/subscribe system", I
>> understand that having a Producer Service which sits in between the
>> Application and the Producer defects the one major purpose of Kafka.
>>
>> So, my question is, How to loosely couple Kafka with my Production
>> Application ?
>>
>> The reason being, I wish to do all producer code and Kafka library
>> maintenance without affecting my large scale Production system. Its not an
>> easy thing to buy a window to these type of changes done on a large scale
>> production application :)
>>
>> Any advice on how this can be achieved(even moderately) will greatly help
>> ?
>>
>> Thanks,
>> Krishna Raj
>>
>
>
Re: Does Kafka Producer service ?
Posted by Jun Rao <ju...@gmail.com>.
Could you be a bit more specific about the issue? As long as there is
protocol compatibility btw the Kafka client and the broker, upgrading the
Kafka client library should be easy, right?
Thanks,
Jun
On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
wrote:
> Hello Amazing Kafka Creators & User,
>
> I have learnt and use kafka in our Production system, so you can count my
> understanding as intermediate.
>
> With the statement that "Kafka has solved the Scalability and Availability
> needs for a large scale message publish/subscribe system", I understand
> that having a Producer Service which sits in between the Application and
> the Producer defects the one major purpose of Kafka.
>
> So, my question is, How to loosely couple Kafka with my Production
> Application ?
>
> The reason being, I wish to do all producer code and Kafka library
> maintenance without affecting my large scale Production system. Its not an
> easy thing to buy a window to these type of changes done on a large scale
> production application :)
>
> Any advice on how this can be achieved(even moderately) will greatly help ?
>
> Thanks,
> Krishna Raj
>
Re: Does Kafka Producer service ?
Posted by Jun Rao <ju...@gmail.com>.
Could you be a bit more specific about the issue? As long as there is
protocol compatibility btw the Kafka client and the broker, upgrading the
Kafka client library should be easy, right?
Thanks,
Jun
On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <re...@gmail.com>
wrote:
> Hello Amazing Kafka Creators & User,
>
> I have learnt and use kafka in our Production system, so you can count my
> understanding as intermediate.
>
> With the statement that "Kafka has solved the Scalability and Availability
> needs for a large scale message publish/subscribe system", I understand
> that having a Producer Service which sits in between the Application and
> the Producer defects the one major purpose of Kafka.
>
> So, my question is, How to loosely couple Kafka with my Production
> Application ?
>
> The reason being, I wish to do all producer code and Kafka library
> maintenance without affecting my large scale Production system. Its not an
> easy thing to buy a window to these type of changes done on a large scale
> production application :)
>
> Any advice on how this can be achieved(even moderately) will greatly help ?
>
> Thanks,
> Krishna Raj
>