You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Shekar Tippur <ct...@gmail.com> on 2015/03/26 16:22:43 UTC
Kafka partition key
Hello,
Want to confirm a basic understanding of Kafka.
If I have a dataset that needs to be partitioned by 4 columns, then the
progression is
{topic1:partition_key1} -> {Group by samza on partition_key1}
->
{topic2:partition_key2} -> {Group by samza on partition_key2}
->
{topic3:partition_key3} -> {Group by samza on partition_key3}
->
{topic4:partition_key4} -> {Group by samza on partition_key4}
Can you please confirm if my understanding is right?
- Shekar
Re: Kafka partition key
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi Shekar,
Please refer to [1]. You can set a custom partitioner through the producer
cofig. You will have to implement your own partitioner based on your
application and partitioning strategy.
Thanks
Milinda
[1] https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
On Thu, Mar 26, 2015 at 2:25 PM, Shekar Tippur <ct...@gmail.com> wrote:
> So if I have a feed with
>
> {user_id:12345,
> ethnicity: asian,
> location: "cerritos, ca",
> Height:"5.9",
> weight: "150 lbs"}
>
> I am referring to https://kafka.apache.org/081/ops.html#topic-config
>
> How do I map the 3 columns - (user_id, ethnicity, and location) to a
> partition id. If I map it this way and say create 10 partitions, each
> partition will contain a subset of data grouped by these columns - right?
>
> - Shekar
>
>
>
>
> On Thu, Mar 26, 2015 at 9:38 AM, Roger Hoover <ro...@gmail.com>
> wrote:
>
> > Hi Richard,
> >
> > You can also partition by a key like "user_id" so that all messages for a
> > given user would end up in the same partition. This can be useful for
> > calculating user-specific aggregations or doing a distributed join where
> > the local state is also partitioned on user_id.
> >
> > Cheers,
> >
> > Roger
> >
> > On Thu, Mar 26, 2015 at 9:28 AM, Richard Lee <rd...@tivo.com> wrote:
> >
> > > Is there a typo below? Are all of these actually in the same topic,
> just
> > > different partitions? Partitioning, AFAIK, is mainly done for
> > parallelism
> > > & throughput reasons. What is the reason for partitioning your dataset
> > by
> > > ‘columns’?
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic
> > > ?
> > >
> > > Richard
> > >
> > > > On Mar 26, 2015, at 8:22 AM, Shekar Tippur <ct...@gmail.com>
> wrote:
> > > >
> > > > Hello,
> > > >
> > > > Want to confirm a basic understanding of Kafka.
> > > > If I have a dataset that needs to be partitioned by 4 columns, then
> the
> > > > progression is
> > > >
> > > > {topic1:partition_key1} -> {Group by samza on partition_key1}
> > > > ->
> > > > {topic2:partition_key2} -> {Group by samza on partition_key2}
> > > > ->
> > > > {topic3:partition_key3} -> {Group by samza on partition_key3}
> > > > ->
> > > > {topic4:partition_key4} -> {Group by samza on partition_key4}
> > > >
> > > > Can you please confirm if my understanding is right?
> > > >
> > > > - Shekar
> > >
> > >
> > > ________________________________
> > >
> > > This email and any attachments may contain confidential and privileged
> > > material for the sole use of the intended recipient. Any review,
> copying,
> > > or distribution of this email (or any attachments) by others is
> > prohibited.
> > > If you are not the intended recipient, please contact the sender
> > > immediately and permanently delete this email and any attachments. No
> > > employee or agent of TiVo Inc. is authorized to conclude any binding
> > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > > Inc. may only be made by a signed written agreement.
> > >
> >
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Kafka partition key
Posted by Shekar Tippur <ct...@gmail.com>.
So if I have a feed with
{user_id:12345,
ethnicity: asian,
location: "cerritos, ca",
Height:"5.9",
weight: "150 lbs"}
I am referring to https://kafka.apache.org/081/ops.html#topic-config
How do I map the 3 columns - (user_id, ethnicity, and location) to a
partition id. If I map it this way and say create 10 partitions, each
partition will contain a subset of data grouped by these columns - right?
- Shekar
On Thu, Mar 26, 2015 at 9:38 AM, Roger Hoover <ro...@gmail.com>
wrote:
> Hi Richard,
>
> You can also partition by a key like "user_id" so that all messages for a
> given user would end up in the same partition. This can be useful for
> calculating user-specific aggregations or doing a distributed join where
> the local state is also partitioned on user_id.
>
> Cheers,
>
> Roger
>
> On Thu, Mar 26, 2015 at 9:28 AM, Richard Lee <rd...@tivo.com> wrote:
>
> > Is there a typo below? Are all of these actually in the same topic, just
> > different partitions? Partitioning, AFAIK, is mainly done for
> parallelism
> > & throughput reasons. What is the reason for partitioning your dataset
> by
> > ‘columns’?
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic
> > ?
> >
> > Richard
> >
> > > On Mar 26, 2015, at 8:22 AM, Shekar Tippur <ct...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > Want to confirm a basic understanding of Kafka.
> > > If I have a dataset that needs to be partitioned by 4 columns, then the
> > > progression is
> > >
> > > {topic1:partition_key1} -> {Group by samza on partition_key1}
> > > ->
> > > {topic2:partition_key2} -> {Group by samza on partition_key2}
> > > ->
> > > {topic3:partition_key3} -> {Group by samza on partition_key3}
> > > ->
> > > {topic4:partition_key4} -> {Group by samza on partition_key4}
> > >
> > > Can you please confirm if my understanding is right?
> > >
> > > - Shekar
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
Re: Kafka partition key
Posted by Roger Hoover <ro...@gmail.com>.
Hi Richard,
You can also partition by a key like "user_id" so that all messages for a
given user would end up in the same partition. This can be useful for
calculating user-specific aggregations or doing a distributed join where
the local state is also partitioned on user_id.
Cheers,
Roger
On Thu, Mar 26, 2015 at 9:28 AM, Richard Lee <rd...@tivo.com> wrote:
> Is there a typo below? Are all of these actually in the same topic, just
> different partitions? Partitioning, AFAIK, is mainly done for parallelism
> & throughput reasons. What is the reason for partitioning your dataset by
> ‘columns’?
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic
> ?
>
> Richard
>
> > On Mar 26, 2015, at 8:22 AM, Shekar Tippur <ct...@gmail.com> wrote:
> >
> > Hello,
> >
> > Want to confirm a basic understanding of Kafka.
> > If I have a dataset that needs to be partitioned by 4 columns, then the
> > progression is
> >
> > {topic1:partition_key1} -> {Group by samza on partition_key1}
> > ->
> > {topic2:partition_key2} -> {Group by samza on partition_key2}
> > ->
> > {topic3:partition_key3} -> {Group by samza on partition_key3}
> > ->
> > {topic4:partition_key4} -> {Group by samza on partition_key4}
> >
> > Can you please confirm if my understanding is right?
> >
> > - Shekar
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
Re: Kafka partition key
Posted by Richard Lee <rd...@tivo.com>.
Is there a typo below? Are all of these actually in the same topic, just different partitions? Partitioning, AFAIK, is mainly done for parallelism & throughput reasons. What is the reason for partitioning your dataset by ‘columns’?
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic?
Richard
> On Mar 26, 2015, at 8:22 AM, Shekar Tippur <ct...@gmail.com> wrote:
>
> Hello,
>
> Want to confirm a basic understanding of Kafka.
> If I have a dataset that needs to be partitioned by 4 columns, then the
> progression is
>
> {topic1:partition_key1} -> {Group by samza on partition_key1}
> ->
> {topic2:partition_key2} -> {Group by samza on partition_key2}
> ->
> {topic3:partition_key3} -> {Group by samza on partition_key3}
> ->
> {topic4:partition_key4} -> {Group by samza on partition_key4}
>
> Can you please confirm if my understanding is right?
>
> - Shekar
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Kafka partition key
Posted by Dotan Patrich <do...@fortscale.com>.
Hi Shekar,
Each kafka partition is basically just a number, you would need to specify
what partitioner strategy to use when mapping your event key to the
partition number.
You can take the 4 columns you have in the event and map it to a partition
number,the partitioner in that case would be a function that would work
similar to that notion: (a, b, c, d) -> (int)
Once you partition your data to different topic partitions, each partition
will hold a sub-set of the dataset that is basically similar to what SQL
"group by" statement would have done.
Hope that helps,
Dotan
On Thu, Mar 26, 2015 at 5:22 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Hello,
>
> Want to confirm a basic understanding of Kafka.
> If I have a dataset that needs to be partitioned by 4 columns, then the
> progression is
>
> {topic1:partition_key1} -> {Group by samza on partition_key1}
> ->
> {topic2:partition_key2} -> {Group by samza on partition_key2}
> ->
> {topic3:partition_key3} -> {Group by samza on partition_key3}
> ->
> {topic4:partition_key4} -> {Group by samza on partition_key4}
>
> Can you please confirm if my understanding is right?
>
> - Shekar
>