You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Barathan Kulothongan <ba...@icloud.com> on 2018/06/25 02:57:21 UTC

Using Kafka as Primary Datastore without compaction

Hi There, I am currently reading the Kafka Definitive guide book as we wanted to Architect an application using Kafka as the primary data store. In the Log Compaction section you have mentioned the dirty records are compacted with leaving just the latest records in the topic. Actually the application we are architecting needs to maintain bi-temporal data with all the message versions of the same key intact. What are the properties we should set in order to make sure none of the messages gets compacted? Or is it always safe to store the messages in another datastore (Postgresql) without relying Kafka as our only data store.

PS: The application that will use Kafka as the datastore is expected to store less than 1GB worth of data in 200+ topics.

Regards,
Barathan Kulothongan

Re: Using Kafka as Primary Datastore without compaction

Posted by Aman Rastogi <am...@gmail.com>.

As per my understanding, you should set the retention period to Long.MAX
hours. This will ensure that your messages won't get compacted because
retention period is huge.

Regards,
Aman

On Mon, Jun 25, 2018 at 8:27 AM, Barathan Kulothongan <barathan_k@icloud.com
> wrote:

> Hi There, I am currently reading the Kafka Definitive guide book as we
> wanted to Architect an application using Kafka as the primary data store.
> In the Log Compaction section you have mentioned the dirty records are
> compacted with leaving just the latest records in the topic. Actually the
> application we are architecting needs to maintain bi-temporal data with all
> the message versions of the same key intact. What are the properties we
> should set in order to make sure none of the messages gets compacted? Or is
> it always safe to store the messages in another datastore (Postgresql)
> without relying Kafka as our only data store.
>
> PS: The application that will use Kafka as the datastore is expected to
> store less than 1GB worth of data in 200+ topics.
>
> Regards,
> Barathan Kulothongan