Copy topic to offline file

["Young, Ben" <Be...@fisglobal.com.INVALID>]

Hi

We have a requirement that we need to be able to copy a portion of a Kafka topic to a file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance on a different machine. This machine may have a different number of partitions etc for the topic. We can't connect them via connect or anything, this is more for offline analysis of data that produce issues in our software, or for providing data for regression tests.

I'm happy to write a tool to do this (I think the OffsetsForTimestamp API does what I need), but does anyone know of any tools out there that do something like this? I've had an initial google, but I can't find anything

Thanks,
Ben

Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.young@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

RE: Copy topic to offline file

["Young, Ben" <Be...@fisglobal.com.INVALID>]

Thanks Francis,

I think it's because of the ad-hoc request nature of what we want. It's not something we always want on. I'll take a look at KSQL

Thanks,
Ben

-----Original Message-----
From: Francis Siefken [mailto:francis@axual.io]
Sent: 08 June 2018 09:21
To: users@kafka.apache.org
Subject: Re: Copy topic to offline file

Hi Ben, you mentioned 'We can't connect them via connect', but connect with it's filestream export and import was my first thought. If you want to export/import time segments you could use KSQL, would this not fulfil your requirement?

Francis

On Thu, Jun 7, 2018 at 7:47 PM, Young, Ben <Be...@fisglobal.com.invalid> wrote:
> Hi
>
> We have a requirement that we need to be able to copy a portion of a Kafka topic to a file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance on a different machine. This machine may have a different number of partitions etc for the topic. We can't connect them via connect or anything, this is more for offline analysis of data that produce issues in our software, or for providing data for regression tests.
>
> I'm happy to write a tool to do this (I think the OffsetsForTimestamp
> API does what I need), but does anyone know of any tools out there
> that do something like this? I've had an initial google, but I can't
> find anything
>
> Thanks,
> Ben
>
> Ben Young
> Principal Software Engineer - Risk
> Risk & Compliance
> T:  +44 (0) 1865 578140
> E: ben.young@fisglobal.com
> FIS | Empowering the Financial World
> FIS Systems Limited * Registered in England and Wales No. 1937159 *
> Registered Office: Level 39, 25 Canada Square, London E14 5LQ
>
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Copy topic to offline file

[Francis Siefken <fr...@axual.io>]

Hi Ben, you mentioned 'We can't connect them via connect', but connect
with it's filestream export and import was my first thought. If you
want to export/import time segments you could use KSQL, would this not
fulfil your requirement?

Francis

On Thu, Jun 7, 2018 at 7:47 PM, Young, Ben
<Be...@fisglobal.com.invalid> wrote:
> Hi
>
> We have a requirement that we need to be able to copy a portion of a Kafka topic to a file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance on a different machine. This machine may have a different number of partitions etc for the topic. We can't connect them via connect or anything, this is more for offline analysis of data that produce issues in our software, or for providing data for regression tests.
>
> I'm happy to write a tool to do this (I think the OffsetsForTimestamp API does what I need), but does anyone know of any tools out there that do something like this? I've had an initial google, but I can't find anything
>
> Thanks,
> Ben
>
> Ben Young
> Principal Software Engineer - Risk
> Risk & Compliance
> T:  +44 (0) 1865 578140
> E: ben.young@fisglobal.com
> FIS | Empowering the Financial World
> FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office: Level 39, 25 Canada Square, London E14 5LQ
>
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.