Re: Does kafka write key to broker?

[Mohit Kathuria <mk...@sprinklr.com>]

Guozhang,

Thanks for the reply. That's exactly what I was looking for.

-Thanks,
Mohit

On Tue, Jun 30, 2015 at 1:52 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi Mohit,
>
> The KeyedMessage has a "key" and "partKey" as it fields, and if the partKey
> is not specified then key will be used as partKey by default.
>
> So for your case, if you do
>
> new KeyedMessage[String, String](topic, null /*key*/, partkey, value)
>
> then the partkey will be used to determine the partition but not written to
> broker.
>
> Guozhang
>
> On Wed, Jun 24, 2015 at 9:45 AM, Mohit Kathuria <mk...@sprinklr.com>
> wrote:
>
> > Thanks Jason.
> >
> > Hi Liquan,
> >
> > I was doing this initially using KeyedMessage without setting the key.
> But
> > if the key is null, the custom partitioner won't be used and kafka will
> > write to random partition which is cached for some time within the
> > producer. I have a custom partitioner to have fairness between the
> > partitions.
> >
> > -Thanks,
> > Mohit
> >
> > On Tue, Jun 23, 2015 at 10:34 PM, Liquan Pei <li...@gmail.com>
> wrote:
> >
> > > Hi Mohit,
> > >
> > > If you instantiate the keyed message with
> > > val topic = "topic"
> > > val value = "value"
> > > val message = new KeyedMessage[String, String](topic,  value);
> > >
> > > Then the key in the KeyedMessage will be null.
> > >
> > > Hope this helps!
> > >
> > > Thanks,
> > > Liquan
> > >
> > > On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <
> mkathuria@sprinklr.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are using kafka 0.8.1.1 in our production cluster. I recently
> > started
> > > > specifying key as the message itself. I just realised that the key is
> > > also
> > > > written to the broker which means that the data is duplicated within
> a
> > > > keyed message. I am going to change the key. Stupid mistake.
> > > >
> > > > However, just out of anxiety, I want to know whether we can turn off
> > > > writing the key to the broker. Any configuration I can change to
> > achieve
> > > > this?
> > > >
> > > > -Thanks,
> > > > Mohit Kathuria
> > > >
> > >
> > >
> > >
> > > --
> > > Liquan Pei
> > > Department of Physics
> > > University of Massachusetts Amherst
> > >
> >
>
>
>
> --
> -- Guozhang
>