You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Craig Alfieri <cr...@antithesis.com> on 2023/09/14 16:38:22 UTC

Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Hello Spark Community-

As part of a research effort, our team here at Antithesis tests for correctness/fault tolerance of major OSS projects.
Our team recently was testing Spark’s Structured Streaming, and we came across a data duplication bug we’d like to work with the teams on to resolve.

Our intention is to utilize this as a future case study for our platform, but prior to doing so we like to have a resolution in place so that an announcement isn’t alarming to the user base.

Attached is a high level .pdf that reviews the High Availability set-up put under test.
This was also tested across the three latest versions, and the same behavior was observed.

We can reproduce this error readily, since our environment is fully deterministic, we are just not Spark experts and would like to work with someone in the community to resolve this.

Please let us know at your earliest convenience.

Best

[signature_2327449931]
Craig Alfieri
c: 917.841.1652
Craig.Alfieri@antithesis.com<ma...@antithesis.com>
New York, NY.
Antithesis.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>

We can't talk about most of the bugs that we've found for our customers,
but some customers like to speak about their work with us:
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis



-- 

*-----------------------------*
*This email and any files transmitted with 
it are confidential and intended solely for the use of the individual or 
entity for whom they are addressed. If you received this message in error, 
please notify the sender and remove it from your system.*


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Craig Alfieri <cr...@antithesis.com>.
Hi Jerry- This is exactly the type of help we're seeking, to confirm the FilestreamSink was not utilized on our test runs.

Our team is going to work towards implementing this and re-running our experiments across the versions.

If everything comes back with similar results, we will reach back out to share more artifacts with this thread.

Thank you Jerry.


From: Jerry Peng <je...@gmail.com>
Date: Thursday, September 14, 2023 at 1:10 PM
To: Craig Alfieri <cr...@antithesis.com>
Cc: user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Craig,

Thank you for bringing this to the community's attention! Do you have any example code you can share that we can use to reproduce this issue?  By the way, how did you determine duplicates in the output?  The FileStreamSink provides exactly-once writes ONLY if you read the output with the FileStreamSource or the FileSource (batch).  A log is used to determine what data is committed or not and those aforementioned sources know how to use that log to read the data "exactly-once".

Best,

Jerry

On Thu, Sep 14, 2023 at 9:48 AM Craig Alfieri <cr...@antithesis.com>> wrote:
Hello Spark Community-

As part of a research effort, our team here at Antithesis tests for correctness/fault tolerance of major OSS projects.
Our team recently was testing Spark’s Structured Streaming, and we came across a data duplication bug we’d like to work with the teams on to resolve.

Our intention is to utilize this as a future case study for our platform, but prior to doing so we like to have a resolution in place so that an announcement isn’t alarming to the user base.

Attached is a high level .pdf that reviews the High Availability set-up put under test.
This was also tested across the three latest versions, and the same behavior was observed.

We can reproduce this error readily, since our environment is fully deterministic, we are just not Spark experts and would like to work with someone in the community to resolve this.

Please let us know at your earliest convenience.

Best

[signature_2327449931]
Craig Alfieri
c: 917.841.1652
Craig.Alfieri@antithesis.com<ma...@antithesis.com>
New York, NY.
Antithesis.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>

We can't talk about most of the bugs that we've found for our customers,
but some customers like to speak about their work with us:
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis



-----------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

-- 

*-----------------------------*
*This email and any files transmitted with 
it are confidential and intended solely for the use of the individual or 
entity for whom they are addressed. If you received this message in error, 
please notify the sender and remove it from your system.*


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Jerry Peng <je...@gmail.com>.
Hi Craig,

Thank you for bringing this to the community's attention! Do you have any
example code you can share that we can use to reproduce this issue?  By the
way, how did you determine duplicates in the output?  The FileStreamSink
provides exactly-once writes ONLY if you read the output with the
FileStreamSource or the FileSource (batch).  A log is used to determine
what data is committed or not and those aforementioned sources know how to
use that log to read the data "exactly-once".

Best,

Jerry

On Thu, Sep 14, 2023 at 9:48 AM Craig Alfieri <cr...@antithesis.com>
wrote:

> Hello Spark Community-
>
>
>
> As part of a research effort, our team here at Antithesis tests for
> correctness/fault tolerance of major OSS projects.
>
> Our team recently was testing Spark’s Structured Streaming, and we came
> across a data duplication bug we’d like to work with the teams on to
> resolve.
>
>
>
> Our intention is to utilize this as a future case study for our platform,
> but prior to doing so we like to have a resolution in place so that an
> announcement isn’t alarming to the user base.
>
>
>
> Attached is a high level .pdf that reviews the High Availability set-up
> put under test.
>
> This was also tested across the three latest versions, and the same
> behavior was observed.
>
>
>
> We can reproduce this error readily, since our environment is fully
> deterministic, we are just not Spark experts and would like to work with
> someone in the community to resolve this.
>
>
>
> Please let us know at your earliest convenience.
>
>
>
> Best
>
>
>
> *[image: signature_2327449931]*
>
> *Craig Alfieri*
>
> c: 917.841.1652
>
> Craig.Alfieri@antithesis.com
>
> New York, NY.
>
> Antithesis.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>
>
>
>
> We can't talk about most of the bugs that we've found for our customers,
>
> but some customers like to speak about their work with us:
>
> https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis
>
>
>
>
>
>
> *-----------------------------*
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>

Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Jerry Peng <je...@gmail.com>.
Craig,

I agree that is something we should improve in the documentation!  Can you
file a JIRA ticket?

Can you rerun the experiment by using the FileStream Source to read the
output to see if there are still duplicates?

On Mon, Sep 25, 2023 at 1:14 PM Craig Alfieri <cr...@antithesis.com>
wrote:

> Hi Jerry –
>
>
>
> Curious your thoughts and where you think we should take things from here.
>
>
>
> Do you feel as though this is documentation related, or should this be
> investigated further?
>
>
>
> Can we be of further assistance?
>
>
>
> Best,
>
>
>
> -Craig
>
>
>
> *From: *Craig Alfieri <Cr...@Antithesis.com>
> *Date: *Tuesday, September 19, 2023 at 12:11 PM
> *To: *Jerry Peng <je...@gmail.com>
> *Cc: *russell.spitzer@gmail.com <ru...@gmail.com>, Mich
> Talebzadeh <mi...@gmail.com>, connor.mckee@antithesis.com <
> connor.mckee@antithesis.com>, user@spark.apache.org <user@spark.apache.org
> >
> *Subject: *Re: Data Duplication Bug Found - Structured Streaming Versions
> 3..4.1, 3.2.4, and 3.3.2
>
> Hi Jerry,
>
>
>
> The team referenced the 3.4.1 documentation
> <https://spark.apache.org/docs/3.4.1/structured-streaming-programming-guide.html#continuous-processing> while
> working through the research effort with Apache Spark Structured Streaming.
> And did read the output files directly to confirm the results of the
> process and determine its reliability. The document states for the File
> Sink method that "Fault-tolerant: Yes (exactly-once)" and no further
> mention of additional actions needed to get the "exactly-once" guarantee.
>
>
>
> This is why we believe this is a potential issue for the Spark community.
>
> The latest 3.5.0 documentation
> <https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html> does
> still state the same guarantees.
>
>
>
> [image: A screenshot of a computer program Description automatically
> generated]
>
>
>
> Let us know your thoughts, eager to help the Spark community where we can.
>
>
>
> Best,
>
>
>
> -Craig
>
>
>
> *From: *Jerry Peng <je...@gmail.com>
> *Date: *Monday, September 18, 2023 at 10:06 PM
> *To: *Craig Alfieri <cr...@antithesis.com>
> *Cc: *russell.spitzer@gmail.com <ru...@gmail.com>, Mich
> Talebzadeh <mi...@gmail.com>, connor.mckee@antithesis.com <
> connor.mckee@antithesis.com>, user@spark.apache.org <user@spark.apache.org
> >
> *Subject: *Re: Data Duplication Bug Found - Structured Streaming Versions
> 3..4.1, 3.2.4, and 3.3.2
>
> Hi Craig,
>
>
>
> Thank you for sending us more information.  Can you answer my previous
> question which I don't think the document addresses. How did you determine
> duplicates in the output?  How was the output data read? The FileStreamSink
> provides exactly-once writes ONLY if you read the output with the
> FileStreamSource or the FileSource (batch).  A log is used to determine
> what data is committed or not and those aforementioned sources know how to
> use that log to read the data "exactly-once".  So there may be duplicated
> data written on disk.  If you simply just read the data files written to
> disk you may see duplicates when there are failures.  However, if you read
> the output location with Spark you should get exactly once results (unless
> there is a bug) since spark will know how to use the commit log to see what
> data files are committed and not.
>
>
>
> Best,
>
>
>
> Jerry
>
>
>
> On Mon, Sep 18, 2023 at 1:18 PM Craig Alfieri <
> craig.alfieri@antithesis.com> wrote:
>
> Hi Russell/Jerry/Mich,
>
>
>
> Appreciate your patience on this.
>
>
>
> Attached are more details on how this duplication “error” was found.
>
> Since we’re still unsure I am using “error” in quotes.
>
>
>
> We’d love the opportunity to work with any of you directly and/or the
> wider Spark community to triage this or get a better understanding of the
> nature of what we’re experiencing.
>
>
>
> Our platform provides the ability to fully reproduce this.
>
>
>
> Once you have had the chance to review the attached draft, let us know if
> there are any questions in the meantime. Again, we welcome the opportunity
> to work with the teams on this.
>
>
>
> Best-
>
> Craig
>
>
>
>
>
>
>
> *From: *Craig Alfieri <Cr...@Antithesis.com>
> *Date: *Thursday, September 14, 2023 at 8:45 PM
> *To: *russell.spitzer@gmail.com <ru...@gmail.com>
> *Cc: *Jerry Peng <je...@gmail.com>, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>, user@spark.apache.org <us...@spark.apache.org>,
> connor.mckee@antithesis.com <co...@antithesis.com>
> *Subject: *Re: Data Duplication Bug Found - Structured Streaming Versions
> 3..4.1, 3.2.4, and 3.3.2
>
> Hi Russell et al,
>
>
>
> Acknowledging receipt; we’ll get these answers back to the group.
>
>
>
> Follow-up forthcoming.
>
>
>
> Craig
>
>
>
>
>
>
>
> On Sep 14, 2023, at 6:38 PM, russell.spitzer@gmail.com wrote:
>
> Exactly once should be output sink dependent, what sink was being used?
>
> Sent from my iPhone
>
>
>
> On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com>
> wrote:
>
> 
>
> Craig,
>
>
>
> Thanks! Please let us know the result!
>
>
>
> Best,
>
>
>
> Jerry
>
>
>
> On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>
>
> Hi Craig,
>
>
>
> Can you please clarify what this bug is and provide sample code causing
> this issue?
>
>
>
> HTH
>
>
> Mich Talebzadeh,
>
> Distinguished Technologist, Solutions Architect & Engineer
>
> London
>
> United Kingdom
>
>
>
>  *Error! Filename not specified.*  view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
>
>
> On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>
> wrote:
>
> Hello Spark Community-
>
>
>
> As part of a research effort, our team here at Antithesis tests for
> correctness/fault tolerance of major OSS projects.
>
> Our team recently was testing Spark’s Structured Streaming, and we came
> across a data duplication bug we’d like to work with the teams on to
> resolve.
>
>
>
> Our intention is to utilize this as a future case study for our platform,
> but prior to doing so we like to have a resolution in place so that an
> announcement isn’t alarming to the user base.
>
>
>
> Attached is a high level .pdf that reviews the High Availability set-up
> put under test.
>
> This was also tested across the three latest versions, and the same
> behavior was observed.
>
>
>
> We can reproduce this error readily, since our environment is fully
> deterministic, we are just not Spark experts and would like to work with
> someone in the community to resolve this.
>
>
>
> Please let us know at your earliest convenience.
>
>
>
> Best
>
>
>
> Error! Filename not specified.
>
> *Craig Alfieri*
>
> c: 917.841.1652
>
> Craig.Alfieri@antithesis.com
>
> New York, NY.
>
> Antithesis.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>
>
>
>
> We can't talk about most of the bugs that we've found for our customers,
>
> but some customers like to speak about their work with us:
>
> https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis
>
>
>
>
>
>
>
> *-----------------------------*
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
> *-----------------------------*
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>
>
>
> *-----------------------------*
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>

Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Craig Alfieri <cr...@antithesis.com>.
Hi Jerry –

Curious your thoughts and where you think we should take things from here.

Do you feel as though this is documentation related, or should this be investigated further?

Can we be of further assistance?

Best,

-Craig

From: Craig Alfieri <Cr...@Antithesis.com>
Date: Tuesday, September 19, 2023 at 12:11 PM
To: Jerry Peng <je...@gmail.com>
Cc: russell.spitzer@gmail.com <ru...@gmail.com>, Mich Talebzadeh <mi...@gmail.com>, connor.mckee@antithesis.com <co...@antithesis.com>, user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Jerry,

The team referenced the 3.4.1 documentation<https://spark.apache.org/docs/3.4.1/structured-streaming-programming-guide.html#continuous-processing> while working through the research effort with Apache Spark Structured Streaming. And did read the output files directly to confirm the results of the process and determine its reliability. The document states for the File Sink method that "Fault-tolerant: Yes (exactly-once)" and no further mention of additional actions needed to get the "exactly-once" guarantee.

This is why we believe this is a potential issue for the Spark community.
The latest 3.5.0 documentation<https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html> does still state the same guarantees.

[A screenshot of a computer program  Description automatically generated]

Let us know your thoughts, eager to help the Spark community where we can.

Best,

-Craig

From: Jerry Peng <je...@gmail.com>
Date: Monday, September 18, 2023 at 10:06 PM
To: Craig Alfieri <cr...@antithesis.com>
Cc: russell.spitzer@gmail.com <ru...@gmail.com>, Mich Talebzadeh <mi...@gmail.com>, connor.mckee@antithesis.com <co...@antithesis.com>, user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Craig,

Thank you for sending us more information.  Can you answer my previous question which I don't think the document addresses. How did you determine duplicates in the output?  How was the output data read? The FileStreamSink provides exactly-once writes ONLY if you read the output with the FileStreamSource or the FileSource (batch).  A log is used to determine what data is committed or not and those aforementioned sources know how to use that log to read the data "exactly-once".  So there may be duplicated data written on disk.  If you simply just read the data files written to disk you may see duplicates when there are failures.  However, if you read the output location with Spark you should get exactly once results (unless there is a bug) since spark will know how to use the commit log to see what data files are committed and not.

Best,

Jerry

On Mon, Sep 18, 2023 at 1:18 PM Craig Alfieri <cr...@antithesis.com>> wrote:
Hi Russell/Jerry/Mich,

Appreciate your patience on this.

Attached are more details on how this duplication “error” was found.
Since we’re still unsure I am using “error” in quotes.

We’d love the opportunity to work with any of you directly and/or the wider Spark community to triage this or get a better understanding of the nature of what we’re experiencing.

Our platform provides the ability to fully reproduce this.

Once you have had the chance to review the attached draft, let us know if there are any questions in the meantime. Again, we welcome the opportunity to work with the teams on this.

Best-
Craig



From: Craig Alfieri <Cr...@Antithesis.com>
Date: Thursday, September 14, 2023 at 8:45 PM
To: russell.spitzer@gmail.com<ma...@gmail.com> <ru...@gmail.com>>
Cc: Jerry Peng <je...@gmail.com>>, Mich Talebzadeh <mi...@gmail.com>>, user@spark.apache.org<ma...@spark.apache.org> <us...@spark.apache.org>>, connor.mckee@antithesis.com<ma...@antithesis.com> <co...@antithesis.com>>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Russell et al,

Acknowledging receipt; we’ll get these answers back to the group.

Follow-up forthcoming.

Craig



On Sep 14, 2023, at 6:38 PM, russell.spitzer@gmail.com<ma...@gmail.com> wrote:
Exactly once should be output sink dependent, what sink was being used?
Sent from my iPhone

On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com>> wrote:

Craig,

Thanks! Please let us know the result!

Best,

Jerry

On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <mi...@gmail.com>> wrote:

Hi Craig,

Can you please clarify what this bug is and provide sample code causing this issue?

HTH

Mich Talebzadeh,
Distinguished Technologist, Solutions Architect & Engineer
London
United Kingdom


 Error! Filename not specified.  view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>

 https://en.everybodywiki.com/Mich_Talebzadeh



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>> wrote:
Hello Spark Community-

As part of a research effort, our team here at Antithesis tests for correctness/fault tolerance of major OSS projects.
Our team recently was testing Spark’s Structured Streaming, and we came across a data duplication bug we’d like to work with the teams on to resolve.

Our intention is to utilize this as a future case study for our platform, but prior to doing so we like to have a resolution in place so that an announcement isn’t alarming to the user base.

Attached is a high level .pdf that reviews the High Availability set-up put under test.
This was also tested across the three latest versions, and the same behavior was observed.

We can reproduce this error readily, since our environment is fully deterministic, we are just not Spark experts and would like to work with someone in the community to resolve this.

Please let us know at your earliest convenience.

Best

Error! Filename not specified.
Craig Alfieri
c: 917.841.1652
Craig.Alfieri@antithesis.com<ma...@antithesis.com>
New York, NY.
Antithesis.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>

We can't talk about most of the bugs that we've found for our customers,
but some customers like to speak about their work with us:
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis



-----------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

-----------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.

-- 

*-----------------------------*
*This email and any files transmitted with 
it are confidential and intended solely for the use of the individual or 
entity for whom they are addressed. If you received this message in error, 
please notify the sender and remove it from your system.*


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Craig Alfieri <cr...@antithesis.com>.
Hi Jerry,

The team referenced the 3.4.1 documentation<https://spark.apache.org/docs/3.4.1/structured-streaming-programming-guide.html#continuous-processing> while working through the research effort with Apache Spark Structured Streaming. And did read the output files directly to confirm the results of the process and determine its reliability. The document states for the File Sink method that "Fault-tolerant: Yes (exactly-once)" and no further mention of additional actions needed to get the "exactly-once" guarantee.

This is why we believe this is a potential issue for the Spark community.
The latest 3.5.0 documentation<https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html> does still state the same guarantees.

[A screenshot of a computer program  Description automatically generated]

Let us know your thoughts, eager to help the Spark community where we can.

Best,

-Craig

From: Jerry Peng <je...@gmail.com>
Date: Monday, September 18, 2023 at 10:06 PM
To: Craig Alfieri <cr...@antithesis.com>
Cc: russell.spitzer@gmail.com <ru...@gmail.com>, Mich Talebzadeh <mi...@gmail.com>, connor.mckee@antithesis.com <co...@antithesis.com>, user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Craig,

Thank you for sending us more information.  Can you answer my previous question which I don't think the document addresses. How did you determine duplicates in the output?  How was the output data read? The FileStreamSink provides exactly-once writes ONLY if you read the output with the FileStreamSource or the FileSource (batch).  A log is used to determine what data is committed or not and those aforementioned sources know how to use that log to read the data "exactly-once".  So there may be duplicated data written on disk.  If you simply just read the data files written to disk you may see duplicates when there are failures.  However, if you read the output location with Spark you should get exactly once results (unless there is a bug) since spark will know how to use the commit log to see what data files are committed and not.

Best,

Jerry

On Mon, Sep 18, 2023 at 1:18 PM Craig Alfieri <cr...@antithesis.com>> wrote:
Hi Russell/Jerry/Mich,

Appreciate your patience on this.

Attached are more details on how this duplication “error” was found.
Since we’re still unsure I am using “error” in quotes.

We’d love the opportunity to work with any of you directly and/or the wider Spark community to triage this or get a better understanding of the nature of what we’re experiencing.

Our platform provides the ability to fully reproduce this.

Once you have had the chance to review the attached draft, let us know if there are any questions in the meantime. Again, we welcome the opportunity to work with the teams on this.

Best-
Craig



From: Craig Alfieri <Cr...@Antithesis.com>
Date: Thursday, September 14, 2023 at 8:45 PM
To: russell.spitzer@gmail.com<ma...@gmail.com> <ru...@gmail.com>>
Cc: Jerry Peng <je...@gmail.com>>, Mich Talebzadeh <mi...@gmail.com>>, user@spark.apache.org<ma...@spark.apache.org> <us...@spark.apache.org>>, connor.mckee@antithesis.com<ma...@antithesis.com> <co...@antithesis.com>>
Subject: Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2
Hi Russell et al,

Acknowledging receipt; we’ll get these answers back to the group.

Follow-up forthcoming.

Craig



On Sep 14, 2023, at 6:38 PM, russell.spitzer@gmail.com<ma...@gmail.com> wrote:
Exactly once should be output sink dependent, what sink was being used?
Sent from my iPhone

On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com>> wrote:

Craig,

Thanks! Please let us know the result!

Best,

Jerry

On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <mi...@gmail.com>> wrote:

Hi Craig,

Can you please clarify what this bug is and provide sample code causing this issue?

HTH

Mich Talebzadeh,
Distinguished Technologist, Solutions Architect & Engineer
London
United Kingdom


 Error! Filename not specified.  view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>

 https://en.everybodywiki.com/Mich_Talebzadeh



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>> wrote:
Hello Spark Community-

As part of a research effort, our team here at Antithesis tests for correctness/fault tolerance of major OSS projects.
Our team recently was testing Spark’s Structured Streaming, and we came across a data duplication bug we’d like to work with the teams on to resolve.

Our intention is to utilize this as a future case study for our platform, but prior to doing so we like to have a resolution in place so that an announcement isn’t alarming to the user base.

Attached is a high level .pdf that reviews the High Availability set-up put under test.
This was also tested across the three latest versions, and the same behavior was observed.

We can reproduce this error readily, since our environment is fully deterministic, we are just not Spark experts and would like to work with someone in the community to resolve this.

Please let us know at your earliest convenience.

Best

Error! Filename not specified.
Craig Alfieri
c: 917.841.1652
Craig.Alfieri@antithesis.com<ma...@antithesis.com>
New York, NY.
Antithesis.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>

We can't talk about most of the bugs that we've found for our customers,
but some customers like to speak about their work with us:
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis



-----------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

-----------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity for whom they are addressed. If you received this message in error, please notify the sender and remove it from your system.

-- 

*-----------------------------*
*This email and any files transmitted with 
it are confidential and intended solely for the use of the individual or 
entity for whom they are addressed. If you received this message in error, 
please notify the sender and remove it from your system.*


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Jerry Peng <je...@gmail.com>.
Hi Craig,

Thank you for sending us more information.  Can you answer my previous
question which I don't think the document addresses. How did you determine
duplicates in the output?  How was the output data read? The FileStreamSink
provides exactly-once writes ONLY if you read the output with the
FileStreamSource or the FileSource (batch).  A log is used to determine
what data is committed or not and those aforementioned sources know how to
use that log to read the data "exactly-once".  So there may be duplicated
data written on disk.  If you simply just read the data files written to
disk you may see duplicates when there are failures.  However, if you read
the output location with Spark you should get exactly once results (unless
there is a bug) since spark will know how to use the commit log to see what
data files are committed and not.

Best,

Jerry

On Mon, Sep 18, 2023 at 1:18 PM Craig Alfieri <cr...@antithesis.com>
wrote:

> Hi Russell/Jerry/Mich,
>
>
>
> Appreciate your patience on this.
>
>
>
> Attached are more details on how this duplication “error” was found.
>
> Since we’re still unsure I am using “error” in quotes.
>
>
>
> We’d love the opportunity to work with any of you directly and/or the
> wider Spark community to triage this or get a better understanding of the
> nature of what we’re experiencing.
>
>
>
> Our platform provides the ability to fully reproduce this.
>
>
>
> Once you have had the chance to review the attached draft, let us know if
> there are any questions in the meantime. Again, we welcome the opportunity
> to work with the teams on this.
>
>
>
> Best-
>
> Craig
>
>
>
>
>
>
>
> *From: *Craig Alfieri <Cr...@Antithesis.com>
> *Date: *Thursday, September 14, 2023 at 8:45 PM
> *To: *russell.spitzer@gmail.com <ru...@gmail.com>
> *Cc: *Jerry Peng <je...@gmail.com>, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>, user@spark.apache.org <us...@spark.apache.org>,
> connor.mckee@antithesis.com <co...@antithesis.com>
> *Subject: *Re: Data Duplication Bug Found - Structured Streaming Versions
> 3..4.1, 3.2.4, and 3.3.2
>
> Hi Russell et al,
>
>
>
> Acknowledging receipt; we’ll get these answers back to the group.
>
>
>
> Follow-up forthcoming.
>
>
>
> Craig
>
>
>
>
>
>
>
> On Sep 14, 2023, at 6:38 PM, russell.spitzer@gmail.com wrote:
>
> Exactly once should be output sink dependent, what sink was being used?
>
> Sent from my iPhone
>
>
>
> On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com>
> wrote:
>
> 
>
> Craig,
>
>
>
> Thanks! Please let us know the result!
>
>
>
> Best,
>
>
>
> Jerry
>
>
>
> On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>
>
> Hi Craig,
>
>
>
> Can you please clarify what this bug is and provide sample code causing
> this issue?
>
>
>
> HTH
>
>
> Mich Talebzadeh,
>
> Distinguished Technologist, Solutions Architect & Engineer
>
> London
>
> United Kingdom
>
>
>
>  [image: Image removed by sender.]  view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
>
>
> On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>
> wrote:
>
> Hello Spark Community-
>
>
>
> As part of a research effort, our team here at Antithesis tests for
> correctness/fault tolerance of major OSS projects.
>
> Our team recently was testing Spark’s Structured Streaming, and we came
> across a data duplication bug we’d like to work with the teams on to
> resolve.
>
>
>
> Our intention is to utilize this as a future case study for our platform,
> but prior to doing so we like to have a resolution in place so that an
> announcement isn’t alarming to the user base.
>
>
>
> Attached is a high level .pdf that reviews the High Availability set-up
> put under test.
>
> This was also tested across the three latest versions, and the same
> behavior was observed.
>
>
>
> We can reproduce this error readily, since our environment is fully
> deterministic, we are just not Spark experts and would like to work with
> someone in the community to resolve this.
>
>
>
> Please let us know at your earliest convenience.
>
>
>
> Best
>
>
>
> Error! Filename not specified.
>
> *Craig Alfieri*
>
> c: 917.841.1652
>
> Craig.Alfieri@antithesis.com
>
> New York, NY.
>
> Antithesis.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>
>
>
>
> We can't talk about most of the bugs that we've found for our customers,
>
> but some customers like to speak about their work with us:
>
> https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis
>
>
>
>
>
>
>
> *-----------------------------*
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
> *-----------------------------*
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>

Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Craig Alfieri <cr...@antithesis.com>.
Hi Russell et al,

  

Acknowledging receipt; we’ll get these answers back to the group.

  

Follow-up forthcoming.

  

Craig

  

  

  

> On Sep 14, 2023, at 6:38 PM, russell.spitzer@gmail.com wrote:  
>  
>

> Exactly once should be output sink dependent, what sink was being used?  
>  
>
>
> Sent from my iPhone
>
>  
>
>

>> On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com>
wrote:  
>  
>
>

>> 

>>

>> Craig,

>>

>>  
>
>>

>> Thanks! Please let us know the result!

>>

>>  
>
>>

>> Best,

>>

>>  
>
>>

>> Jerry

>>

>>  
>
>>

>> On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh
<[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:  
>
>>

>>>  
>
>>>

>>> Hi Craig,

>>>

>>>  
>
>>>

>>> Can you please clarify what this bug is and provide sample code causing
this issue?

>>>

>>>  
>
>>>

>>> HTH

>>>

>>>  
>
>>>

>>> Mich Talebzadeh,

>>>

>>> Distinguished Technologist, Solutions Architect & Engineer  
>
>>>

>>> London  
>
>>>

>>> United Kingdom

>>>

>>>  
>
>>>

>>>  ![](https://ci3.googleusercontent.com/mail-
sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE)
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)

>>>

>>>  
>
>>>

>>>  <https://en.everybodywiki.com/Mich_Talebzadeh>

>>>

>>>  
>>>

>>>  **Disclaimer:**  Use it at your own risk. Any and all responsibility for
any loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed. The
author will in no case be liable for any monetary damages arising from such
loss, damage or destruction.

>>>

>>>  
>>>

>>>  
>
>>>

>>>  
>
>>>

>>> On Thu, 14 Sept 2023 at 17:48, Craig Alfieri
<[craig.alfieri@antithesis.com](mailto:craig.alfieri@antithesis.com)> wrote:  
>
>>>

>>>> Hello Spark Community- __ __

>>>>

>>>> __  __

>>>>

>>>> As part of a research effort, our team here at Antithesis tests for
correctness/fault tolerance of major OSS projects. __ __

>>>>

>>>> Our team recently was testing Spark’s Structured Streaming, and we came
across a data duplication bug we’d like to work with the teams on to resolve.
__ __

>>>>

>>>> __  __

>>>>

>>>> Our intention is to utilize this as a future case study for our platform,
but prior to doing so we like to have a resolution in place so that an
announcement isn’t alarming to the user base. __ __

>>>>

>>>> __  __

>>>>

>>>> Attached is a high level .pdf that reviews the High Availability set-up
put under test. __ __

>>>>

>>>> This was also tested across the three latest versions, and the same
behavior was observed. __ __

>>>>

>>>> __  __

>>>>

>>>> We can reproduce this error readily, since our environment is fully
deterministic, we are just not Spark experts and would like to work with
someone in the community to resolve this. __ __

>>>>

>>>> __  __

>>>>

>>>> Please let us know at your earliest convenience. __ __

>>>>

>>>> __  __

>>>>

>>>> Best __ __

>>>>

>>>> __  __

>>>>

>>>> ******____**

>>>>

>>>> |

>>>>

>>>> **Craig Alfieri __ __**

>>>>

>>>> |

>>>>

>>>> c: 917.841.1652 __ __

>>>>

>>>> [Craig.Alfieri@antithesis.com](mailto:Craig.Alfieri@antithesis.com) __ __

>>>>

>>>> |

>>>>

>>>> New York, NY. __ __

>>>>

>>>>
[Antithesis.com](https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-
GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=) __ __  
>>>>  
>>>> ---|---|---|---  
>>>>  
>>>> __  __

>>>>

>>>> We can't talk about most of the bugs that we've found for our customers,
____

>>>>

>>>> but some customers like to speak about their work with us: __ __

>>>>

>>>> [ https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-
Antithesis](https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-
Antithesis) __ __

>>>>

>>>> __  __

>>>>

>>>> __  __

>>>>

>>>>  
>  
>
>>>>

>>>> _\-----------------------------_

>>>>

>>>>  _This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity for whom they are
addressed. If you received this message in error, please notify the sender and
remove it from your system._  
>
>>>>

>>>>  
>  \---------------------------------------------------------------------  
>  To unsubscribe e-mail: [user-unsubscribe@spark.apache.org](mailto:user-
> unsubscribe@spark.apache.org)
>>>

>>>  
>  \---------------------------------------------------------------------  
>  To unsubscribe e-mail: [user-unsubscribe@spark.apache.org](mailto:user-
> unsubscribe@spark.apache.org)

  
  

_\-----------------------------_

 _This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity for whom they are addressed. If
you received this message in error, please notify the sender and remove it
from your system._  

\--------------------------------------------------------------------- To
unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by ru...@gmail.com.
Exactly once should be output sink dependent, what sink was being used?  
  

Sent from my iPhone

  

> On Sep 14, 2023, at 4:52 PM, Jerry Peng <je...@gmail.com> wrote:  
>  
>

> 
>
> Craig,
>
>  
>
>
> Thanks! Please let us know the result!
>
>  
>
>
> Best,
>
>  
>
>
> Jerry
>
>  
>
>
> On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh
> <[mich.talebzadeh@gmail.com](mailto:mich.talebzadeh@gmail.com)> wrote:  
>
>

>>  
>
>>

>> Hi Craig,

>>

>>  
>
>>

>> Can you please clarify what this bug is and provide sample code causing
this issue?

>>

>>  
>
>>

>> HTH

>>

>>  
>
>>

>> Mich Talebzadeh,

>>

>> Distinguished Technologist, Solutions Architect & Engineer  
>
>>

>> London  
>
>>

>> United Kingdom

>>

>>  
>
>>

>>  ![](https://ci3.googleusercontent.com/mail-
sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE)
**** [ view my Linkedin profile](https://www.linkedin.com/in/mich-talebzadeh-
ph-d-5205b2/)

>>

>>  
>
>>

>>  <https://en.everybodywiki.com/Mich_Talebzadeh>

>>

>>  
>>

>>  **Disclaimer:**  Use it at your own risk. Any and all responsibility for
any loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed. The
author will in no case be liable for any monetary damages arising from such
loss, damage or destruction.

>>

>>  
>>

>>  
>
>>

>>  
>
>>

>> On Thu, 14 Sept 2023 at 17:48, Craig Alfieri
<[craig.alfieri@antithesis.com](mailto:craig.alfieri@antithesis.com)> wrote:  
>
>>

>>> Hello Spark Community- __ __

>>>

>>> __  __

>>>

>>> As part of a research effort, our team here at Antithesis tests for
correctness/fault tolerance of major OSS projects. __ __

>>>

>>> Our team recently was testing Spark’s Structured Streaming, and we came
across a data duplication bug we’d like to work with the teams on to resolve.
__ __

>>>

>>> __  __

>>>

>>> Our intention is to utilize this as a future case study for our platform,
but prior to doing so we like to have a resolution in place so that an
announcement isn’t alarming to the user base. __ __

>>>

>>> __  __

>>>

>>> Attached is a high level .pdf that reviews the High Availability set-up
put under test. __ __

>>>

>>> This was also tested across the three latest versions, and the same
behavior was observed. __ __

>>>

>>> __  __

>>>

>>> We can reproduce this error readily, since our environment is fully
deterministic, we are just not Spark experts and would like to work with
someone in the community to resolve this. __ __

>>>

>>> __  __

>>>

>>> Please let us know at your earliest convenience. __ __

>>>

>>> __  __

>>>

>>> Best __ __

>>>

>>> __  __

>>>

>>> ******____**

>>>

>>> |

>>>

>>> **Craig Alfieri __ __**

>>>

>>> |

>>>

>>> c: 917.841.1652 __ __

>>>

>>> [Craig.Alfieri@antithesis.com](mailto:Craig.Alfieri@antithesis.com) __ __

>>>

>>> |

>>>

>>> New York, NY. __ __

>>>

>>>
[Antithesis.com](https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-
GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=) __ __  
>>>  
>>> ---|---|---|---  
>>>  
>>> __  __

>>>

>>> We can't talk about most of the bugs that we've found for our customers,
____

>>>

>>> but some customers like to speak about their work with us: __ __

>>>

>>> [ https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-
Antithesis](https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-
Antithesis) __ __

>>>

>>> __  __

>>>

>>> __  __

>>>

>>>  
>  
>
>>>

>>> _\-----------------------------_

>>>

>>>  _This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity for whom they are
addressed. If you received this message in error, please notify the sender and
remove it from your system._  
>
>>>

>>>  
>  \---------------------------------------------------------------------  
>  To unsubscribe e-mail: [user-unsubscribe@spark.apache.org](mailto:user-
> unsubscribe@spark.apache.org)
>>

>>  
>  \---------------------------------------------------------------------  
>  To unsubscribe e-mail: [user-unsubscribe@spark.apache.org](mailto:user-
> unsubscribe@spark.apache.org)


Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Jerry Peng <je...@gmail.com>.
Craig,

Thanks! Please let us know the result!

Best,

Jerry

On Thu, Sep 14, 2023 at 12:22 PM Mich Talebzadeh <mi...@gmail.com>
wrote:

>
> Hi Craig,
>
> Can you please clarify what this bug is and provide sample code causing
> this issue?
>
> HTH
>
> Mich Talebzadeh,
> Distinguished Technologist, Solutions Architect & Engineer
> London
> United Kingdom
>
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>
> wrote:
>
>> Hello Spark Community-
>>
>>
>>
>> As part of a research effort, our team here at Antithesis tests for
>> correctness/fault tolerance of major OSS projects.
>>
>> Our team recently was testing Spark’s Structured Streaming, and we came
>> across a data duplication bug we’d like to work with the teams on to
>> resolve.
>>
>>
>>
>> Our intention is to utilize this as a future case study for our platform,
>> but prior to doing so we like to have a resolution in place so that an
>> announcement isn’t alarming to the user base.
>>
>>
>>
>> Attached is a high level .pdf that reviews the High Availability set-up
>> put under test.
>>
>> This was also tested across the three latest versions, and the same
>> behavior was observed.
>>
>>
>>
>> We can reproduce this error readily, since our environment is fully
>> deterministic, we are just not Spark experts and would like to work with
>> someone in the community to resolve this.
>>
>>
>>
>> Please let us know at your earliest convenience.
>>
>>
>>
>> Best
>>
>>
>>
>> *[image: signature_2327449931]*
>>
>> *Craig Alfieri*
>>
>> c: 917.841.1652
>>
>> Craig.Alfieri@antithesis.com
>>
>> New York, NY.
>>
>> Antithesis.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>
>>
>>
>>
>> We can't talk about most of the bugs that we've found for our customers,
>>
>> but some customers like to speak about their work with us:
>>
>> https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis
>>
>>
>>
>>
>>
>>
>> *-----------------------------*
>> *This email and any files transmitted with it are confidential and
>> intended solely for the use of the individual or entity for whom they are
>> addressed. If you received this message in error, please notify the sender
>> and remove it from your system.*
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Data Duplication Bug Found - Structured Streaming Versions 3..4.1, 3.2.4, and 3.3.2

Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Craig,

Can you please clarify what this bug is and provide sample code causing
this issue?

HTH

Mich Talebzadeh,
Distinguished Technologist, Solutions Architect & Engineer
London
United Kingdom


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>


 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 14 Sept 2023 at 17:48, Craig Alfieri <cr...@antithesis.com>
wrote:

> Hello Spark Community-
>
>
>
> As part of a research effort, our team here at Antithesis tests for
> correctness/fault tolerance of major OSS projects.
>
> Our team recently was testing Spark’s Structured Streaming, and we came
> across a data duplication bug we’d like to work with the teams on to
> resolve.
>
>
>
> Our intention is to utilize this as a future case study for our platform,
> but prior to doing so we like to have a resolution in place so that an
> announcement isn’t alarming to the user base.
>
>
>
> Attached is a high level .pdf that reviews the High Availability set-up
> put under test.
>
> This was also tested across the three latest versions, and the same
> behavior was observed.
>
>
>
> We can reproduce this error readily, since our environment is fully
> deterministic, we are just not Spark experts and would like to work with
> someone in the community to resolve this.
>
>
>
> Please let us know at your earliest convenience.
>
>
>
> Best
>
>
>
> *[image: signature_2327449931]*
>
> *Craig Alfieri*
>
> c: 917.841.1652
>
> Craig.Alfieri@antithesis.com
>
> New York, NY.
>
> Antithesis.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.antithesis.com_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=1FbSpGgVIpZO4QkQDmXk7jc1BFVciZWVioOvdJ86ubY&m=5SVjNvtYuy6icWSaP0lwjzTQw1Cc7JQO9QVaxn5KxqTdH8HC1HHURutlp5rgiaMH&s=SRmgBE5ImnGZ-GuqL3X6Q_6NPYiay1gLRbcUUofPIHo&e=>
>
>
>
> We can't talk about most of the bugs that we've found for our customers,
>
> but some customers like to speak about their work with us:
>
> https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis
>
>
>
>
>
>
> *-----------------------------*
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity for whom they are
> addressed. If you received this message in error, please notify the sender
> and remove it from your system.*
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>