linkedin and pageview producer + when kafka is down

[S Ahmed <sa...@gmail.com>]

I'd love to get some insights on how things work at linkedin in terms of
your web servers and kafka producers.

You guys probably connect to multiple kafka clusters, so let's assume you
are only connecting to a single cluster.

1. do you use a single producer for all message types/topics?

2. For your pageview topic i.e. it is getting sent on a per page request
(albiet it is batched):

*What happens when your kafka cluster is down?  Will your web application
behave as normal or will it really slow things down?

Locally on my laptop I shutdown my vagrant that is running kafka, and the
page renders very slow when the producer is down.

Or do you use some smart circuit breaker logic that will stop trying to
send producer messages if kafka is down?

Re: linkedin and pageview producer + when kafka is down

[Neha Narkhede <ne...@gmail.com>]

1. do you use a single producer for all message types/topics?

Not really. It is typical for a java service/process to have multiple
producers sending to different topics

2. For your pageview topic i.e. it is getting sent on a per page request
(albiet it is batched):

*What happens when your kafka cluster is down?  Will your web application
behave as normal or will it really slow things down?

It wouldn't slow down the web application. It will drop data if it's
internal in memory queue fills up and notifies the service of the loss
through a metric



On Mon, Jun 16, 2014 at 10:10 AM, S Ahmed <sa...@gmail.com> wrote:

> I'd love to get some insights on how things work at linkedin in terms of
> your web servers and kafka producers.
>
> You guys probably connect to multiple kafka clusters, so let's assume you
> are only connecting to a single cluster.
>
> 1. do you use a single producer for all message types/topics?
>
> 2. For your pageview topic i.e. it is getting sent on a per page request
> (albiet it is batched):
>
> *What happens when your kafka cluster is down?  Will your web application
> behave as normal or will it really slow things down?
>
> Locally on my laptop I shutdown my vagrant that is running kafka, and the
> page renders very slow when the producer is down.
>
> Or do you use some smart circuit breaker logic that will stop trying to
> send producer messages if kafka is down?
>