You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tecno Brain <ce...@gmail.com> on 2022/11/30 14:35:53 UTC
Copying data from one topic into another
Hi,
I'm using Kafka 2.8
I have a topic with an original retention of 1 day with a large amount of
messages.
We had a problem with some messages being corrupted and we decided to
skip about 3.5 hours of messages; but not all of them are corrupted. So, we
would like to process this period of time again with a modified version of
our consumer application.
In order to avoid the messages being deleted, we extended the retention to
3 days.
But we are now again close to have the messages being deleted and we
cannot extend the retention anymore (we would go over the limit of the disk
space).
So, I would like to extract the messages between the offsets that cover
the 3.5h
Is there a quick way to extract these messages and copy them into another
topic ?
(After that, I want to set the retention back to the original 1 day)
Would kcat help?
Re: Copying data from one topic into another
Posted by Schehrzade <sc...@gmail.com>.
This is kind of boring I’m sorry I subscribed but I’ll keep them because I
might learn from you
On Wed, Nov 30, 2022 at 9:41 AM Tecno Brain <ce...@gmail.com>
wrote:
> By the way, we use Avro for the messages.
>
>
>
> On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain <ce...@gmail.com>
> wrote:
>
> > Hi,
> > I'm using Kafka 2.8
> > I have a topic with an original retention of 1 day with a large amount
> > of messages.
> >
> > We had a problem with some messages being corrupted and we decided to
> > skip about 3.5 hours of messages; but not all of them are corrupted. So,
> we
> > would like to process this period of time again with a modified version
> of
> > our consumer application.
> >
> > In order to avoid the messages being deleted, we extended the retention
> > to 3 days.
> > But we are now again close to have the messages being deleted and we
> > cannot extend the retention anymore (we would go over the limit of the
> disk
> > space).
> >
> > So, I would like to extract the messages between the offsets that cover
> > the 3.5h
> >
> > Is there a quick way to extract these messages and copy them into
> > another topic ?
> > (After that, I want to set the retention back to the original 1 day)
> >
> > Would kcat help?
> >
> >
> >
> >
> >
>
Re: Copying data from one topic into another
Posted by Schehrzade <sc...@gmail.com>.
I like it i was just kidding I’ll subscribe there too for sure!! I was just
adoringly teasing you all 😛
On Wed, Nov 30, 2022 at 12:04 PM nagpavan chilakam <
nagpavan.chilakam@gmail.com> wrote:
> Maybe you can even explore `kafka-console-consumer` to get the messages
> from a certain offset
>
> On Wed, Nov 30, 2022 at 8:11 PM Tecno Brain <ce...@gmail.com>
> wrote:
>
> > By the way, we use Avro for the messages.
> >
> >
> >
> > On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain <
> cerebrotecnologico@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I'm using Kafka 2.8
> > > I have a topic with an original retention of 1 day with a large
> amount
> > > of messages.
> > >
> > > We had a problem with some messages being corrupted and we decided to
> > > skip about 3.5 hours of messages; but not all of them are corrupted.
> So,
> > we
> > > would like to process this period of time again with a modified version
> > of
> > > our consumer application.
> > >
> > > In order to avoid the messages being deleted, we extended the
> retention
> > > to 3 days.
> > > But we are now again close to have the messages being deleted and we
> > > cannot extend the retention anymore (we would go over the limit of the
> > disk
> > > space).
> > >
> > > So, I would like to extract the messages between the offsets that
> cover
> > > the 3.5h
> > >
> > > Is there a quick way to extract these messages and copy them into
> > > another topic ?
> > > (After that, I want to set the retention back to the original 1 day)
> > >
> > > Would kcat help?
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Copying data from one topic into another
Posted by nagpavan chilakam <na...@gmail.com>.
Maybe you can even explore `kafka-console-consumer` to get the messages
from a certain offset
On Wed, Nov 30, 2022 at 8:11 PM Tecno Brain <ce...@gmail.com>
wrote:
> By the way, we use Avro for the messages.
>
>
>
> On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain <ce...@gmail.com>
> wrote:
>
> > Hi,
> > I'm using Kafka 2.8
> > I have a topic with an original retention of 1 day with a large amount
> > of messages.
> >
> > We had a problem with some messages being corrupted and we decided to
> > skip about 3.5 hours of messages; but not all of them are corrupted. So,
> we
> > would like to process this period of time again with a modified version
> of
> > our consumer application.
> >
> > In order to avoid the messages being deleted, we extended the retention
> > to 3 days.
> > But we are now again close to have the messages being deleted and we
> > cannot extend the retention anymore (we would go over the limit of the
> disk
> > space).
> >
> > So, I would like to extract the messages between the offsets that cover
> > the 3.5h
> >
> > Is there a quick way to extract these messages and copy them into
> > another topic ?
> > (After that, I want to set the retention back to the original 1 day)
> >
> > Would kcat help?
> >
> >
> >
> >
> >
>
Re: Copying data from one topic into another
Posted by Tecno Brain <ce...@gmail.com>.
By the way, we use Avro for the messages.
On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain <ce...@gmail.com>
wrote:
> Hi,
> I'm using Kafka 2.8
> I have a topic with an original retention of 1 day with a large amount
> of messages.
>
> We had a problem with some messages being corrupted and we decided to
> skip about 3.5 hours of messages; but not all of them are corrupted. So, we
> would like to process this period of time again with a modified version of
> our consumer application.
>
> In order to avoid the messages being deleted, we extended the retention
> to 3 days.
> But we are now again close to have the messages being deleted and we
> cannot extend the retention anymore (we would go over the limit of the disk
> space).
>
> So, I would like to extract the messages between the offsets that cover
> the 3.5h
>
> Is there a quick way to extract these messages and copy them into
> another topic ?
> (After that, I want to set the retention back to the original 1 day)
>
> Would kcat help?
>
>
>
>
>