Is kafka message timestamp preserved in mirror maker

[tao xiao <xi...@gmail.com>]

Hi,

As per the description in KIP-32 the timestamp of Kafka message is
unchanged mirrored from one cluster to another if createTime is used. But I
tested with mirror maker in Kafka-0.10 this doesn't seem the case. The
timestamp of the same message is different in source and target. I checked
the latest code from trunk the timestamp field is not set in the producer
record that is sent to target.

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L678

Is this a bug or preserving timestamp is not a built-in feature?

Re: Is kafka message timestamp preserved in mirror maker

[Gerard Klijs <ge...@dizzit.com>]

Although I think it should have been an included option, it's very easy to
create and use your own message handler with the mirror maker. You can
simply copy the timestamp and type from the consumerecord to the
producerecord.

On Wed, Jun 1, 2016 at 5:48 PM Gwen Shapira <gw...@confluent.io> wrote:

> The intent was definitely as you described, but I think we forgot to
> actually modify the code accordingly.
>
> Do you mind opening a JIRA on the issue?
>
> Gwen
>
> On Wed, Jun 1, 2016 at 4:13 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Hi,
> >
> > As per the description in KIP-32 the timestamp of Kafka message is
> > unchanged mirrored from one cluster to another if createTime is used.
> But I
> > tested with mirror maker in Kafka-0.10 this doesn't seem the case. The
> > timestamp of the same message is different in source and target. I
> checked
> > the latest code from trunk the timestamp field is not set in the producer
> > record that is sent to target.
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L678
> >
> > Is this a bug or preserving timestamp is not a built-in feature?
> >
>

Re: Is kafka message timestamp preserved in mirror maker

[Gwen Shapira <gw...@confluent.io>]

The intent was definitely as you described, but I think we forgot to
actually modify the code accordingly.

Do you mind opening a JIRA on the issue?

Gwen

On Wed, Jun 1, 2016 at 4:13 PM, tao xiao <xi...@gmail.com> wrote:

> Hi,
>
> As per the description in KIP-32 the timestamp of Kafka message is
> unchanged mirrored from one cluster to another if createTime is used. But I
> tested with mirror maker in Kafka-0.10 this doesn't seem the case. The
> timestamp of the same message is different in source and target. I checked
> the latest code from trunk the timestamp field is not set in the producer
> record that is sent to target.
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L678
>
> Is this a bug or preserving timestamp is not a built-in feature?
>