You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Gnanasoundari Soundarajan <gn...@man-es.com> on 2022/03/08 16:31:46 UTC

Spark kafka structured streaming - how to prevent dataloss

Hi,

In spark, it uses checkpoints to keep track of offsets in kafka. If there is any data loss, can we edit the file and reduce the data loss? Please suggest the best practices to reduce the data loss under exceptional scenarios.

Regards,
Gnana

RE: Spark kafka structured streaming - how to prevent dataloss

Posted by Gnanasoundari Soundarajan <gn...@man-es.com>.

Hi all,

Any suggestion?

Regards,
Gnana

From: Gnanasoundari Soundarajan <gn...@man-es.com>
Sent: Tuesday, March 8, 2022 10:02 PM
To: user@spark.apache.org
Subject: Spark kafka structured streaming - how to prevent dataloss

Hi,

In spark, it uses checkpoints to keep track of offsets in kafka. If there is any data loss, can we edit the file and reduce the data loss? Please suggest the best practices to reduce the data loss under exceptional scenarios.

Regards,
Gnana