You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Weston Pace <we...@gmail.com> on 2021/05/19 05:37:07 UTC

[Discuss] [Proposal] [C++] Arrow multithreaded stress test suite

I spoke a while ago about working on a multithreaded stress test
suite.  I have put together some very early details[1].  I would
appreciate any feedback.

The goal would be to stress test the C++ dataset API (and soon C++
execution plans and perhaps someday a language independent logical
plan / query language).  This would happen both through concurrent
execution and through injected failure (e.g. filesystem failures).
The goal would primarily be the detection of segmentation faults and
deadlock.

One particular item I could use feedback on is how this gets deployed.
In my mind this would be an ongoing test that is continuously running
against the previous nightly build.  Such a test would quickly consume
Apache's GHA minutes so I don't think GHA is an option.  Other free CI
options probably wouldn't have enough minutes for a continuous daily
test (e.g. ~40k minutes).  Fortunately, solving this is not a
precursor to developing it :).

-Weston

[1] https://docs.google.com/document/d/1Ozq_TxKvQ7F2XH9Mea_ORefDyKm_EtzRHd948KHCgsg/edit?usp=sharing

Re: [Discuss] [Proposal] [C++] Arrow multithreaded stress test suite

Posted by Weston Pace <we...@gmail.com>.
> I would recommend writing such tests in Python, such as is already done
> for the CSV reader.

Agreed, that is my current thinking as well.

> I'm not sure what you have in mind.  You're intending to run this test
> 40k minutes per day?

40k minutes per month.  24 hours * 60 minutes * 30 days ~ 40k minutes.

> Such an approach
> can be performed by introducing explicit breakpoints in the code and
> starting when the breakpoint is reached the other code that we know might
> cause problems when executed concurrently.

We have some number of tests doing this in async_generator_test.cc.  I
agree that it is a good idea in general.  Here though I'm after issues
that we don't know about and so I wound't know where to insert the
breakpoints.  Once an issue has been identified and fixed then a more
targeted regression unit test could be created using these techniques.

On Tue, May 18, 2021 at 10:23 PM Alessandro Molina
<al...@ursacomputing.com> wrote:
>
> Another approach that could reduce the amount of heavy tests that we have
> to write (if the tests are written in Python) might be to drive the code to
> interleave in the ways we feel might introduce problems. Such an approach
> can be performed by introducing explicit breakpoints in the code and
> starting when the breakpoint is reached the other code that we know might
> cause problems when executed concurrently.
>
> For example, Imagine you want to simulate what happens when two threads
> write to a file concurrently, you put a Breakpoint on file.write, wait for
> that breakpoint to be reached, and explicitly invoke another file.write
> when that happens.
>
> That way instead of having to throw tons of threads trying to trigger for
> race conditions randomly you can reduce the amount of time/computation by
> explicitly searching for problems in the areas that are more certain to
> hide them and raising the chances that they happen by forcing the two code
> blocks to always run interleaved in the way you expect that might cause
> problems.
>
> mock.patch to wrap the function where you want to stop is the easy way to
> put those breakpoints in Python usually. At crunch for example was written
> a Breakpoint class extensively used to write tests that simulate race
> conditions which was released under MIT license (
> https://github.com/Crunch-io/diagnose#breakpoints )
>
> On Wed, May 19, 2021 at 9:01 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Le 19/05/2021 à 07:37, Weston Pace a écrit :
> > > I spoke a while ago about working on a multithreaded stress test
> > > suite.  I have put together some very early details[1].  I would
> > > appreciate any feedback.
> >
> > I would recommend writing such tests in Python, such as is already done
> > for the CSV reader.
> >
> > > One particular item I could use feedback on is how this gets deployed.
> > > In my mind this would be an ongoing test that is continuously running
> > > against the previous nightly build.  Such a test would quickly consume
> > > Apache's GHA minutes so I don't think GHA is an option.  Other free CI
> > > options probably wouldn't have enough minutes for a continuous daily
> > > test (e.g. ~40k minutes).
> >
> > I'm not sure what you have in mind.  You're intending to run this test
> > 40k minutes per day?
> >
> > Regards
> >
> > Antoine.
> >

Re: [Discuss] [Proposal] [C++] Arrow multithreaded stress test suite

Posted by Alessandro Molina <al...@ursacomputing.com>.
Another approach that could reduce the amount of heavy tests that we have
to write (if the tests are written in Python) might be to drive the code to
interleave in the ways we feel might introduce problems. Such an approach
can be performed by introducing explicit breakpoints in the code and
starting when the breakpoint is reached the other code that we know might
cause problems when executed concurrently.

For example, Imagine you want to simulate what happens when two threads
write to a file concurrently, you put a Breakpoint on file.write, wait for
that breakpoint to be reached, and explicitly invoke another file.write
when that happens.

That way instead of having to throw tons of threads trying to trigger for
race conditions randomly you can reduce the amount of time/computation by
explicitly searching for problems in the areas that are more certain to
hide them and raising the chances that they happen by forcing the two code
blocks to always run interleaved in the way you expect that might cause
problems.

mock.patch to wrap the function where you want to stop is the easy way to
put those breakpoints in Python usually. At crunch for example was written
a Breakpoint class extensively used to write tests that simulate race
conditions which was released under MIT license (
https://github.com/Crunch-io/diagnose#breakpoints )

On Wed, May 19, 2021 at 9:01 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 19/05/2021 à 07:37, Weston Pace a écrit :
> > I spoke a while ago about working on a multithreaded stress test
> > suite.  I have put together some very early details[1].  I would
> > appreciate any feedback.
>
> I would recommend writing such tests in Python, such as is already done
> for the CSV reader.
>
> > One particular item I could use feedback on is how this gets deployed.
> > In my mind this would be an ongoing test that is continuously running
> > against the previous nightly build.  Such a test would quickly consume
> > Apache's GHA minutes so I don't think GHA is an option.  Other free CI
> > options probably wouldn't have enough minutes for a continuous daily
> > test (e.g. ~40k minutes).
>
> I'm not sure what you have in mind.  You're intending to run this test
> 40k minutes per day?
>
> Regards
>
> Antoine.
>

Re: [Discuss] [Proposal] [C++] Arrow multithreaded stress test suite

Posted by Antoine Pitrou <an...@python.org>.
Le 19/05/2021 à 07:37, Weston Pace a écrit :
> I spoke a while ago about working on a multithreaded stress test
> suite.  I have put together some very early details[1].  I would
> appreciate any feedback.

I would recommend writing such tests in Python, such as is already done 
for the CSV reader.

> One particular item I could use feedback on is how this gets deployed.
> In my mind this would be an ongoing test that is continuously running
> against the previous nightly build.  Such a test would quickly consume
> Apache's GHA minutes so I don't think GHA is an option.  Other free CI
> options probably wouldn't have enough minutes for a continuous daily
> test (e.g. ~40k minutes).

I'm not sure what you have in mind.  You're intending to run this test 
40k minutes per day?

Regards

Antoine.