Adding partitions...

[Niclas Hedhman <ni...@hedhman.org>]

Hi,

we are trying to figure out how to layout our topics and partitions. And
one thing I can't find in the documentation is;

What happens to data that is sitting in the 'old partitions' when I add a
new one?

My gut feeling says that everything remains in the partitions as they are,
and if we need to have partitions associated with keys, we have to
implement this re-shuffling ourselves, i.e. read all the messages out and
put them back in.

Is it so? And is there any hooks to assist in this?


Cheers
-- 
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug

Re: Adding partitions...

[Niclas Hedhman <ni...@hedhman.org>]

Right... We don't yet know if there is going to be "meaning" to the
partition (such as content affinity of consumers has been discussed), I am
simply setting the stage of how things really work.


Thanks
Niclas


On Fri, May 23, 2014 at 2:41 PM, svante karlsson <sa...@csi.se> wrote:

> No reshuffeling will take place. And reading messages and put them back in
> again will not remove the messages from their "old" partition so the same
> message will the exist in more than one partition - eventually to get aged
> out of the oldest partion.
>
> If you use partitioning to distribute the load then it's possible to just
> leave the old data in the existing partitions and let newer data fill all
> the partitions.
>
> If the partitioning means something to your applications then you can
> always create a new topic and populate that from the old topic. This will
> double your data for some time before you can age out the old topic.
>
>
> 2014-05-23 12:44 GMT+02:00 Niclas Hedhman <ni...@hedhman.org>:
>
> > Hi,
> >
> > we are trying to figure out how to layout our topics and partitions. And
> > one thing I can't find in the documentation is;
> >
> > What happens to data that is sitting in the 'old partitions' when I add a
> > new one?
> >
> > My gut feeling says that everything remains in the partitions as they
> are,
> > and if we need to have partitions associated with keys, we have to
> > implement this re-shuffling ourselves, i.e. read all the messages out and
> > put them back in.
> >
> > Is it so? And is there any hooks to assist in this?
> >
> >
> > Cheers
> > --
> > Niclas Hedhman, Software Developer
> > 河南南路555弄15号1901室。
> > http://www.qi4j.org - New Energy for Java
> >
> > I live here; http://tinyurl.com/3xugrbk
> > I work here; http://tinyurl.com/6a2pl4j
> > I relax here; http://tinyurl.com/2cgsug
> >
>



-- 
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug

Re: Adding partitions...

[svante karlsson <sa...@csi.se>]

No reshuffeling will take place. And reading messages and put them back in
again will not remove the messages from their "old" partition so the same
message will the exist in more than one partition - eventually to get aged
out of the oldest partion.

If you use partitioning to distribute the load then it's possible to just
leave the old data in the existing partitions and let newer data fill all
the partitions.

If the partitioning means something to your applications then you can
always create a new topic and populate that from the old topic. This will
double your data for some time before you can age out the old topic.


2014-05-23 12:44 GMT+02:00 Niclas Hedhman <ni...@hedhman.org>:

> Hi,
>
> we are trying to figure out how to layout our topics and partitions. And
> one thing I can't find in the documentation is;
>
> What happens to data that is sitting in the 'old partitions' when I add a
> new one?
>
> My gut feeling says that everything remains in the partitions as they are,
> and if we need to have partitions associated with keys, we have to
> implement this re-shuffling ourselves, i.e. read all the messages out and
> put them back in.
>
> Is it so? And is there any hooks to assist in this?
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> 河南南路555弄15号1901室。
> http://www.qi4j.org - New Energy for Java
>
> I live here; http://tinyurl.com/3xugrbk
> I work here; http://tinyurl.com/6a2pl4j
> I relax here; http://tinyurl.com/2cgsug
>