You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mohit Kathuria <mk...@sprinklr.com> on 2015/06/23 17:18:37 UTC
Does kafka write key to broker?
Hi,
We are using kafka 0.8.1.1 in our production cluster. I recently started
specifying key as the message itself. I just realised that the key is also
written to the broker which means that the data is duplicated within a
keyed message. I am going to change the key. Stupid mistake.
However, just out of anxiety, I want to know whether we can turn off
writing the key to the broker. Any configuration I can change to achieve
this?
-Thanks,
Mohit Kathuria
Re: Does kafka write key to broker?
Posted by Mohit Kathuria <mk...@sprinklr.com>.
Guozhang,
Thanks for the reply. That's exactly what I was looking for.
-Thanks,
Mohit
On Tue, Jun 30, 2015 at 1:52 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Mohit,
>
> The KeyedMessage has a "key" and "partKey" as it fields, and if the partKey
> is not specified then key will be used as partKey by default.
>
> So for your case, if you do
>
> new KeyedMessage[String, String](topic, null /*key*/, partkey, value)
>
> then the partkey will be used to determine the partition but not written to
> broker.
>
> Guozhang
>
> On Wed, Jun 24, 2015 at 9:45 AM, Mohit Kathuria <mk...@sprinklr.com>
> wrote:
>
> > Thanks Jason.
> >
> > Hi Liquan,
> >
> > I was doing this initially using KeyedMessage without setting the key.
> But
> > if the key is null, the custom partitioner won't be used and kafka will
> > write to random partition which is cached for some time within the
> > producer. I have a custom partitioner to have fairness between the
> > partitions.
> >
> > -Thanks,
> > Mohit
> >
> > On Tue, Jun 23, 2015 at 10:34 PM, Liquan Pei <li...@gmail.com>
> wrote:
> >
> > > Hi Mohit,
> > >
> > > If you instantiate the keyed message with
> > > val topic = "topic"
> > > val value = "value"
> > > val message = new KeyedMessage[String, String](topic, value);
> > >
> > > Then the key in the KeyedMessage will be null.
> > >
> > > Hope this helps!
> > >
> > > Thanks,
> > > Liquan
> > >
> > > On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <
> mkathuria@sprinklr.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are using kafka 0.8.1.1 in our production cluster. I recently
> > started
> > > > specifying key as the message itself. I just realised that the key is
> > > also
> > > > written to the broker which means that the data is duplicated within
> a
> > > > keyed message. I am going to change the key. Stupid mistake.
> > > >
> > > > However, just out of anxiety, I want to know whether we can turn off
> > > > writing the key to the broker. Any configuration I can change to
> > achieve
> > > > this?
> > > >
> > > > -Thanks,
> > > > Mohit Kathuria
> > > >
> > >
> > >
> > >
> > > --
> > > Liquan Pei
> > > Department of Physics
> > > University of Massachusetts Amherst
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Does kafka write key to broker?
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mohit,
The KeyedMessage has a "key" and "partKey" as it fields, and if the partKey
is not specified then key will be used as partKey by default.
So for your case, if you do
new KeyedMessage[String, String](topic, null /*key*/, partkey, value)
then the partkey will be used to determine the partition but not written to
broker.
Guozhang
On Wed, Jun 24, 2015 at 9:45 AM, Mohit Kathuria <mk...@sprinklr.com>
wrote:
> Thanks Jason.
>
> Hi Liquan,
>
> I was doing this initially using KeyedMessage without setting the key. But
> if the key is null, the custom partitioner won't be used and kafka will
> write to random partition which is cached for some time within the
> producer. I have a custom partitioner to have fairness between the
> partitions.
>
> -Thanks,
> Mohit
>
> On Tue, Jun 23, 2015 at 10:34 PM, Liquan Pei <li...@gmail.com> wrote:
>
> > Hi Mohit,
> >
> > If you instantiate the keyed message with
> > val topic = "topic"
> > val value = "value"
> > val message = new KeyedMessage[String, String](topic, value);
> >
> > Then the key in the KeyedMessage will be null.
> >
> > Hope this helps!
> >
> > Thanks,
> > Liquan
> >
> > On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <mk...@sprinklr.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We are using kafka 0.8.1.1 in our production cluster. I recently
> started
> > > specifying key as the message itself. I just realised that the key is
> > also
> > > written to the broker which means that the data is duplicated within a
> > > keyed message. I am going to change the key. Stupid mistake.
> > >
> > > However, just out of anxiety, I want to know whether we can turn off
> > > writing the key to the broker. Any configuration I can change to
> achieve
> > > this?
> > >
> > > -Thanks,
> > > Mohit Kathuria
> > >
> >
> >
> >
> > --
> > Liquan Pei
> > Department of Physics
> > University of Massachusetts Amherst
> >
>
--
-- Guozhang
Re: Does kafka write key to broker?
Posted by Mohit Kathuria <mk...@sprinklr.com>.
Thanks Jason.
Hi Liquan,
I was doing this initially using KeyedMessage without setting the key. But
if the key is null, the custom partitioner won't be used and kafka will
write to random partition which is cached for some time within the
producer. I have a custom partitioner to have fairness between the
partitions.
-Thanks,
Mohit
On Tue, Jun 23, 2015 at 10:34 PM, Liquan Pei <li...@gmail.com> wrote:
> Hi Mohit,
>
> If you instantiate the keyed message with
> val topic = "topic"
> val value = "value"
> val message = new KeyedMessage[String, String](topic, value);
>
> Then the key in the KeyedMessage will be null.
>
> Hope this helps!
>
> Thanks,
> Liquan
>
> On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <mk...@sprinklr.com>
> wrote:
>
> > Hi,
> >
> > We are using kafka 0.8.1.1 in our production cluster. I recently started
> > specifying key as the message itself. I just realised that the key is
> also
> > written to the broker which means that the data is duplicated within a
> > keyed message. I am going to change the key. Stupid mistake.
> >
> > However, just out of anxiety, I want to know whether we can turn off
> > writing the key to the broker. Any configuration I can change to achieve
> > this?
> >
> > -Thanks,
> > Mohit Kathuria
> >
>
>
>
> --
> Liquan Pei
> Department of Physics
> University of Massachusetts Amherst
>
Re: Does kafka write key to broker?
Posted by Liquan Pei <li...@gmail.com>.
Hi Mohit,
If you instantiate the keyed message with
val topic = "topic"
val value = "value"
val message = new KeyedMessage[String, String](topic, value);
Then the key in the KeyedMessage will be null.
Hope this helps!
Thanks,
Liquan
On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <mk...@sprinklr.com>
wrote:
> Hi,
>
> We are using kafka 0.8.1.1 in our production cluster. I recently started
> specifying key as the message itself. I just realised that the key is also
> written to the broker which means that the data is duplicated within a
> keyed message. I am going to change the key. Stupid mistake.
>
> However, just out of anxiety, I want to know whether we can turn off
> writing the key to the broker. Any configuration I can change to achieve
> this?
>
> -Thanks,
> Mohit Kathuria
>
--
Liquan Pei
Department of Physics
University of Massachusetts Amherst
Re: Does kafka write key to broker?
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Mohit,
Unfortunately, I don't think there's any such configuration.
By the way, there are some pretty cool things you can do with keys in Kafka
(such as semantic partitioning and log compaction). I don't know if they
would help in your use case, but it might be worth checking out
http://kafka.apache.org/documentation.html#design for more details.
-Jason
On Tue, Jun 23, 2015 at 8:18 AM, Mohit Kathuria <mk...@sprinklr.com>
wrote:
> Hi,
>
> We are using kafka 0.8.1.1 in our production cluster. I recently started
> specifying key as the message itself. I just realised that the key is also
> written to the broker which means that the data is duplicated within a
> keyed message. I am going to change the key. Stupid mistake.
>
> However, just out of anxiety, I want to know whether we can turn off
> writing the key to the broker. Any configuration I can change to achieve
> this?
>
> -Thanks,
> Mohit Kathuria
>