You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "bharath kumar avusherla (JIRA)" <ji...@apache.org> on 2018/08/07 22:41:00 UTC

[jira] [Created] (SPARK-25052) Is there any possibility that spark structured streaming generate duplicates in the output?

bharath kumar avusherla created SPARK-25052:
-----------------------------------------------

             Summary: Is there any possibility that spark structured streaming generate duplicates in the output?
                 Key: SPARK-25052
                 URL: https://issues.apache.org/jira/browse/SPARK-25052
             Project: Spark
          Issue Type: Question
          Components: Spark Core
    Affects Versions: 2.3.0
            Reporter: bharath kumar avusherla


We recently observed that the spark structured streaming generated duplicates in the output when reading from Kafka topic and storing the output to the S3 (and checkpointing in S3).  We ran into this issue twice. This is not reproducible. Is there anyone has ever faced this kind of issue before? Is this because of S3 eventual consistency?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org