You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by "An, Hongguo (CORP)" <Ho...@ADP.com.INVALID> on 2023/05/02 19:48:37 UTC

MM2 -- failed to copy old data

Hi:
When I am running MM2 (3.4.0), any message older than 1 days are not copying, the topic has retention period as 5 days.
Got error:

org.apache.kafka.common.errors.InvalidTimestampException: Timestamp 1682664968019 of message with offset 0 is out of range. The timestamp should be within [1682965965321, 1683138765321]

Please help

Thanks
Andrew



This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Re: MM2 -- failed to copy old data

Posted by "An, Hongguo (CORP)" <Ho...@ADP.com.INVALID>.

Thanks Greg. That seems the cause -- this is the default settings from MSK

On 5/2/23, 1:23 PM, "Greg Harris" <greg.harris@aiven.io.INVA <ma...@aiven.io.INVA>LID> wrote:

WARNING: Do not click links or open attachments unless you recognize the source of the email and know the contents are safe.

**********************************************************************
Andrew,

The broker appears to be rejecting the timestamp of the replicated record,
which is the same as the source record timestamp in MM2. I think you will
need to relax the timestamp validation, which is controlled by these
configurations:

https://urldefense.com/v3/__https://kafka.apache.org/documentation/ <https://urldefense.com/v3/__https://kafka.apache.org/documentation/>*topicconfigs_message.timestamp.difference.max.ms__;Iw!!L8-7AA!SdFgn2zlm9XVUm7DdGbsGJPS9NHYxYXONzI8-GCGiJzpAWZpTX6TPHha9LIIedjicL2L0mxVILn4VJlkx9cgXGv0YXI$
https://urldefense.com/v3/__https://kafka.apache.org/documentation/ <https://urldefense.com/v3/__https://kafka.apache.org/documentation/>*brokerconfigs_log.message.timestamp.difference.max.ms__;Iw!!L8-7AA!SdFgn2zlm9XVUm7DdGbsGJPS9NHYxYXONzI8-GCGiJzpAWZpTX6TPHha9LIIedjicL2L0mxVILn4VJlkx9cgzLFI1yQ$

I believe that if you want 5-day old timestamps to be preserved, that this
configuration must be at least 5 days either globally or just for your
destination topic.

I hope this helps!
Greg Harris

On Tue, May 2, 2023 at 12:49 PM An, Hongguo (CORP)
<Hongguo.An@adp.com.inva <ma...@adp.com.inva>lid> wrote:

> Hi:
> When I am running MM2 (3.4.0), any message older than 1 days are not
> copying, the topic has retention period as 5 days.
> Got error:
>
> org.apache.kafka.common.errors.InvalidTimestampException: Timestamp
> 1682664968019 of message with offset 0 is out of range. The timestamp
> should be within [1682965965321, 1683138765321]
>
> Please help
>
> Thanks
> Andrew
>
>
>
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>

This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Re: MM2 -- failed to copy old data

Posted by Greg Harris <gr...@aiven.io.INVALID>.

Andrew,

The broker appears to be rejecting the timestamp of the replicated record,
which is the same as the source record timestamp in MM2. I think you will
need to relax the timestamp validation, which is controlled by these
configurations:

https://kafka.apache.org/documentation/#topicconfigs_message.timestamp.difference.max.ms
https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.difference.max.ms

I believe that if you want 5-day old timestamps to be preserved, that this
configuration must be at least 5 days either globally or just for your
destination topic.

I hope this helps!
Greg Harris

On Tue, May 2, 2023 at 12:49 PM An, Hongguo (CORP)
<Ho...@adp.com.invalid> wrote:

> Hi:
> When I am running MM2 (3.4.0), any message older than 1 days are not
> copying, the topic has retention period as 5 days.
> Got error:
>
> org.apache.kafka.common.errors.InvalidTimestampException: Timestamp
> 1682664968019 of message with offset 0 is out of range. The timestamp
> should be within [1682965965321, 1683138765321]
>
> Please help
>
> Thanks
> Andrew
>
>
>
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>