You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Alex Petrov <al...@coffeenco.de> on 2023/12/21 21:22:54 UTC

Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Hey folks,

I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch. 

Good news for everyone that it: 
  a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
  b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
  c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.

Since we were effectively relying on this code for a while now, and the intention now is to converge to: 
  a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
  b) a testing tool that can be useful for both trivial cases, and complex scenarios 
myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.

I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.

For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.

I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty. 

You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.

Thank you and happy upcoming holidays,
--Alex

[1] https://issues.apache.org/jira/browse/CASSANDRA-19210
[2] https://issues.apache.org/jira/browse/CASSANDRA-18932

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Francisco Guerrero <fr...@apache.org>.
+1 thanks for this effort! 

On 2023/12/21 21:22:54 Alex Petrov wrote:
> Hey folks,
> 
> I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch. 
> 
> Good news for everyone that it: 
>   a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.
> 
> Since we were effectively relying on this code for a while now, and the intention now is to converge to: 
>   a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex scenarios 
> myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.
> 
> I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
> 
> For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
> 
> I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty. 
> 
> You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.
> 
> Thank you and happy upcoming holidays,
> --Alex
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Alex Petrov <al...@coffeenco.de>.
I did mean the md file, which explains all internal intricacies. Also there is a blog post [1] 

My plan was to:
 1. Introduce easily copiable samples 
 2. add more Java doc
 3. Talk to other contributors and collect information about missing pieces / how to make it more accessible 
I might ask for help when we have some better understanding of what we want to achieve.

Thank you 
—Alex

[1] https://cassandra.apache.org/_/blog/Harry-an-Open-Source-Fuzz-Testing-and-Verification-Tool-for-Apache-Cassandra.html

On Tue, Jan 2, 2024, at 10:54 PM, Lorina Poland wrote:
> Is there any user-facing documentation (for developers) that should be added? I note that you say there is "extensive documentation"; I presume that you are referring to the README.md in the repo?
> 
> If there is a desire to add documentation to the website, as opposed to the MD files in the repo, please reach out to me.
> 
> Thanks,
> Lorina
> 
> On 2023/12/21 21:22:54 Alex Petrov wrote:
> > Hey folks,
> > 
> > I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch. 
> > 
> > Good news for everyone that it: 
> >   a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
> >   b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
> >   c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.
> > 
> > Since we were effectively relying on this code for a while now, and the intention now is to converge to: 
> >   a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
> >   b) a testing tool that can be useful for both trivial cases, and complex scenarios 
> > myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.
> > 
> > I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
> > 
> > For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
> > 
> > I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty. 
> > 
> > You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.
> > 
> > Thank you and happy upcoming holidays,
> > --Alex
> > 
> > [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> > [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
> > 
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Lorina Poland <po...@apache.org>.
Is there any user-facing documentation (for developers) that should be added? I note that you say there is "extensive documentation"; I presume that you are referring to the README.md in the repo?

If there is a desire to add documentation to the website, as opposed to the MD files in the repo, please reach out to me.

Thanks,
Lorina

On 2023/12/21 21:22:54 Alex Petrov wrote:
> Hey folks,
> 
> I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch. 
> 
> Good news for everyone that it: 
>   a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.
> 
> Since we were effectively relying on this code for a while now, and the intention now is to converge to: 
>   a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex scenarios 
> myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.
> 
> I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
> 
> For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
> 
> I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty. 
> 
> You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.
> 
> Thank you and happy upcoming holidays,
> --Alex
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by German Eichberger via dev <de...@cassandra.apache.org>.
+1
________________________________
From: Patrick McFadin <pm...@gmail.com>
Sent: Friday, December 22, 2023 9:12 AM
To: dev@cassandra.apache.org <de...@cassandra.apache.org>
Subject: [EXTERNAL] Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

It was great having some more extended discussions about Harry in person last week. Anything we can do to make it easier for anyone to test Cassandra thoroughly is an easy +1 from me!

Thanks for all your efforts so far, Alex.

Patrick

On Fri, Dec 22, 2023 at 8:03 AM Jacek Lewandowski <le...@gmail.com>> wrote:
Obviously +1

Thank you Alex

pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti <su...@gmail.com>> napisał:
+1, thank you for your efforts in bringing Harry in-tree. Anything that improves the testing ecosystem for Cassandra, particularly around complex scenarios / edge cases  goes a long way in improving reliability, and with having a powerful tool like Harry in-tree, it is a lot more accessible to the developers than it has been. Also, thank you for keeping in mind the onboarding experience of developers.

- Sumanth

On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov <al...@coffeenco.de>> wrote:
Some follow-up tickets to establish the project direction:

https://issues.apache.org/jira/browse/CASSANDRA-19229

Two other things that we will work on in Tree are:
https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM test for partition-restricted 2i queries)
https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI read and write fuzz test)

If you would like to get your recently added feature tested with Harry model, please let me know!

On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
+1

Sounds like a great change that will help us unify around a common testing paradigm, and even pave the path to in-tree load testing plus integrated correctness checking which would be extremely valuable!

-Joey

On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <ca...@gmail.com>> wrote:
+1

Agree w/ all the justifications mentioned above.

As a reviewer on CASSANDRA-19210<https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were to a.) look at the directory, naming, and package structure of the ported code, b.) make sure IDE integration was working, and c.) make sure any modifications to existing code (rather than direct code movements from cassandra-harry) were straightforward.

On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de>> wrote:

Hey folks,

I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch.

Good news for everyone that it:
  a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
  b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
  c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.

Since we were effectively relying on this code for a while now, and the intention now is to converge to:
  a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
  b) a testing tool that can be useful for both trivial cases, and complex scenarios
myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.

I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.

For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.

I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty.

You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.

Thank you and happy upcoming holidays,
--Alex

[1] https://issues.apache.org/jira/browse/CASSANDRA-19210
[2] https://issues.apache.org/jira/browse/CASSANDRA-18932



Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Patrick McFadin <pm...@gmail.com>.
It was great having some more extended discussions about Harry in person
last week. Anything we can do to make it easier for anyone to test
Cassandra thoroughly is an easy +1 from me!

Thanks for all your efforts so far, Alex.

Patrick

On Fri, Dec 22, 2023 at 8:03 AM Jacek Lewandowski <
lewandowski.jacek@gmail.com> wrote:

> Obviously +1
>
> Thank you Alex
>
> pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti <
> sumanth.pasupuleti.is@gmail.com> napisał:
>
>> +1, thank you for your efforts in bringing Harry in-tree. Anything that
>> improves the testing ecosystem for Cassandra, particularly around complex
>> scenarios / edge cases  goes a long way in improving reliability, and with
>> having a powerful tool like Harry in-tree, it is a lot more accessible to
>> the developers than it has been. Also, thank you for keeping in mind the
>> onboarding experience of developers.
>>
>> - Sumanth
>>
>> On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov <al...@coffeenco.de> wrote:
>>
>>> Some follow-up tickets to establish the project direction:
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-19229
>>>
>>> Two other things that we will work on in Tree are:
>>> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
>>> test for partition-restricted 2i queries)
>>> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded
>>> SAI read and write fuzz test)
>>>
>>> If you would like to get your recently added feature tested with Harry
>>> model, please let me know!
>>>
>>> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>>>
>>> +1
>>>
>>> Sounds like a great change that will help us unify around a common
>>> testing paradigm, and even pave the path to in-tree load testing plus
>>> integrated correctness checking which would be extremely valuable!
>>>
>>> -Joey
>>>
>>> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <
>>> calebrackliffe@gmail.com> wrote:
>>>
>>> +1
>>>
>>> Agree w/ all the justifications mentioned above.
>>>
>>> As a reviewer on CASSANDRA-19210
>>> <https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were
>>> to a.) look at the directory, naming, and package structure of the ported
>>> code, b.) make sure IDE integration was working, and c.) make sure any
>>> modifications to existing code (rather than direct code movements from
>>> cassandra-harry) were straightforward.
>>>
>>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> Hey folks,
>>>
>>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>>> trigger one more CI run overnight, and my intention was to merge it some
>>> time soon, but I wanted to give a fair warning here, since this is a
>>> relatively large patch.
>>>
>>> Good news for everyone that it:
>>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>>> namely) code that was using Harry already.
>>>   b) the only tests that are changed are ones that used a duplicate
>>> version of placement simulator we had both for testing TCM, and in Harry
>>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>>> have some base for examples/usage.
>>>
>>> Since we were effectively relying on this code for a while now, and the
>>> intention now is to converge to:
>>>   a) fewer different generators, and have a shareable version of
>>> generators for everyone to use accross the base
>>>   b) a testing tool that can be useful for both trivial cases, and
>>> complex scenarios
>>> myself and many other Cassandra contributors have expressed an opinion
>>> that bringing Harry in-tree will be highly benefitial.
>>>
>>> I strongly believe that bringing Harry in-tree will help to lower the
>>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>>> Previously, it has been rather difficult to debug edge cases because I had
>>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>>> time consuming. Moreover, I believe we have missed at very least one RT
>>> regression [2] because Harry was not in-tree, as its tests would've caught
>>> the issue even with the model that existed.
>>>
>>> For other recently found issues, I think having Harry in-tree would have
>>> substantially lowered a turnaround time, and allowed me to share repros
>>> with developers of corresponding features much quicker.
>>>
>>> I do expect a slight learning curve for Harry, but my intention is to
>>> build a web of simple tests (worked on some of them yesterday after
>>> conversation with David already), which can follow the in-jvm-dtest pattern
>>> of find-similar-test / copy / modify. There's already copious
>>> documentation, so I do not believe not having docs for Harry was ever an
>>> issue, since there have been plenty.
>>>
>>> You all are aware of my dedication to testing and quality of Apache
>>> Cassandra, and I hope you also see the benefits of having a model checker
>>> in-tree.
>>>
>>> Thank you and happy upcoming holidays,
>>> --Alex
>>>
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>>
>>>
>>>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Jacek Lewandowski <le...@gmail.com>.
Obviously +1

Thank you Alex

pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti <
sumanth.pasupuleti.is@gmail.com> napisał:

> +1, thank you for your efforts in bringing Harry in-tree. Anything that
> improves the testing ecosystem for Cassandra, particularly around complex
> scenarios / edge cases  goes a long way in improving reliability, and with
> having a powerful tool like Harry in-tree, it is a lot more accessible to
> the developers than it has been. Also, thank you for keeping in mind the
> onboarding experience of developers.
>
> - Sumanth
>
> On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov <al...@coffeenco.de> wrote:
>
>> Some follow-up tickets to establish the project direction:
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-19229
>>
>> Two other things that we will work on in Tree are:
>> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
>> test for partition-restricted 2i queries)
>> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded
>> SAI read and write fuzz test)
>>
>> If you would like to get your recently added feature tested with Harry
>> model, please let me know!
>>
>> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>>
>> +1
>>
>> Sounds like a great change that will help us unify around a common
>> testing paradigm, and even pave the path to in-tree load testing plus
>> integrated correctness checking which would be extremely valuable!
>>
>> -Joey
>>
>> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <ca...@gmail.com>
>> wrote:
>>
>> +1
>>
>> Agree w/ all the justifications mentioned above.
>>
>> As a reviewer on CASSANDRA-19210
>> <https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were
>> to a.) look at the directory, naming, and package structure of the ported
>> code, b.) make sure IDE integration was working, and c.) make sure any
>> modifications to existing code (rather than direct code movements from
>> cassandra-harry) were straightforward.
>>
>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:
>>
>>
>> Hey folks,
>>
>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>> trigger one more CI run overnight, and my intention was to merge it some
>> time soon, but I wanted to give a fair warning here, since this is a
>> relatively large patch.
>>
>> Good news for everyone that it:
>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>> namely) code that was using Harry already.
>>   b) the only tests that are changed are ones that used a duplicate
>> version of placement simulator we had both for testing TCM, and in Harry
>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>> have some base for examples/usage.
>>
>> Since we were effectively relying on this code for a while now, and the
>> intention now is to converge to:
>>   a) fewer different generators, and have a shareable version of
>> generators for everyone to use accross the base
>>   b) a testing tool that can be useful for both trivial cases, and
>> complex scenarios
>> myself and many other Cassandra contributors have expressed an opinion
>> that bringing Harry in-tree will be highly benefitial.
>>
>> I strongly believe that bringing Harry in-tree will help to lower the
>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>> Previously, it has been rather difficult to debug edge cases because I had
>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>> time consuming. Moreover, I believe we have missed at very least one RT
>> regression [2] because Harry was not in-tree, as its tests would've caught
>> the issue even with the model that existed.
>>
>> For other recently found issues, I think having Harry in-tree would have
>> substantially lowered a turnaround time, and allowed me to share repros
>> with developers of corresponding features much quicker.
>>
>> I do expect a slight learning curve for Harry, but my intention is to
>> build a web of simple tests (worked on some of them yesterday after
>> conversation with David already), which can follow the in-jvm-dtest pattern
>> of find-similar-test / copy / modify. There's already copious
>> documentation, so I do not believe not having docs for Harry was ever an
>> issue, since there have been plenty.
>>
>> You all are aware of my dedication to testing and quality of Apache
>> Cassandra, and I hope you also see the benefits of having a model checker
>> in-tree.
>>
>> Thank you and happy upcoming holidays,
>> --Alex
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>
>>
>>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Sumanth Pasupuleti <su...@gmail.com>.
+1, thank you for your efforts in bringing Harry in-tree. Anything that
improves the testing ecosystem for Cassandra, particularly around complex
scenarios / edge cases  goes a long way in improving reliability, and with
having a powerful tool like Harry in-tree, it is a lot more accessible to
the developers than it has been. Also, thank you for keeping in mind the
onboarding experience of developers.

- Sumanth

On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov <al...@coffeenco.de> wrote:

> Some follow-up tickets to establish the project direction:
>
> https://issues.apache.org/jira/browse/CASSANDRA-19229
>
> Two other things that we will work on in Tree are:
> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
> test for partition-restricted 2i queries)
> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI
> read and write fuzz test)
>
> If you would like to get your recently added feature tested with Harry
> model, please let me know!
>
> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>
> +1
>
> Sounds like a great change that will help us unify around a common testing
> paradigm, and even pave the path to in-tree load testing plus integrated
> correctness checking which would be extremely valuable!
>
> -Joey
>
> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <ca...@gmail.com>
> wrote:
>
> +1
>
> Agree w/ all the justifications mentioned above.
>
> As a reviewer on CASSANDRA-19210
> <https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were to
> a.) look at the directory, naming, and package structure of the ported
> code, b.) make sure IDE integration was working, and c.) make sure any
> modifications to existing code (rather than direct code movements from
> cassandra-harry) were straightforward.
>
> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:
>
>
> Hey folks,
>
> I am mostly done with a patch that brings Harry in-tree [1]. I will
> trigger one more CI run overnight, and my intention was to merge it some
> time soon, but I wanted to give a fair warning here, since this is a
> relatively large patch.
>
> Good news for everyone that it:
>   a) touches no production code whatsoever. Only test (in-jvm dtest
> namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate
> version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to
> have some base for examples/usage.
>
> Since we were effectively relying on this code for a while now, and the
> intention now is to converge to:
>   a) fewer different generators, and have a shareable version of
> generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex
> scenarios
> myself and many other Cassandra contributors have expressed an opinion
> that bringing Harry in-tree will be highly benefitial.
>
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>
> I do expect a slight learning curve for Harry, but my intention is to
> build a web of simple tests (worked on some of them yesterday after
> conversation with David already), which can follow the in-jvm-dtest pattern
> of find-similar-test / copy / modify. There's already copious
> documentation, so I do not believe not having docs for Harry was ever an
> issue, since there have been plenty.
>
> You all are aware of my dedication to testing and quality of Apache
> Cassandra, and I hope you also see the benefits of having a model checker
> in-tree.
>
> Thank you and happy upcoming holidays,
> --Alex
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>
>
>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Alex Petrov <al...@coffeenco.de>.
Some follow-up tickets to establish the project direction:

https://issues.apache.org/jira/browse/CASSANDRA-19229

Two other things that we will work on in Tree are:
https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM test for partition-restricted 2i queries) 
https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI read and write fuzz test)

If you would like to get your recently added feature tested with Harry model, please let me know!

On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
> +1
> 
> Sounds like a great change that will help us unify around a common testing paradigm, and even pave the path to in-tree load testing plus integrated correctness checking which would be extremely valuable!
> 
> -Joey
> 
> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <ca...@gmail.com> wrote:
>> +1
>> 
>> Agree w/ all the justifications mentioned above.
>> 
>> As a reviewer on CASSANDRA-19210 <https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were to a.) look at the directory, naming, and package structure of the ported code, b.) make sure IDE integration was working, and c.) make sure any modifications to existing code (rather than direct code movements from cassandra-harry) were straightforward.
>> 
>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:
>>> __
>>> Hey folks,
>>> 
>>> I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one more CI run overnight, and my intention was to merge it some time soon, but I wanted to give a fair warning here, since this is a relatively large patch. 
>>> 
>>> Good news for everyone that it: 
>>>   a) touches no production code whatsoever. Only test (in-jvm dtest namely) code that was using Harry already.
>>>   b) the only tests that are changed are ones that used a duplicate version of placement simulator we had both for testing TCM, and in Harry
>>>   c) in addition, I have converted 3 existing TCM tests to a new API to have some base for examples/usage.
>>> 
>>> Since we were effectively relying on this code for a while now, and the intention now is to converge to: 
>>>   a) fewer different generators, and have a shareable version of generators for everyone to use accross the base
>>>   b) a testing tool that can be useful for both trivial cases, and complex scenarios 
>>> myself and many other Cassandra contributors have expressed an opinion that bringing Harry in-tree will be highly benefitial.
>>> 
>>> I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
>>> 
>>> For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
>>> 
>>> I do expect a slight learning curve for Harry, but my intention is to build a web of simple tests (worked on some of them yesterday after conversation with David already), which can follow the in-jvm-dtest pattern of find-similar-test / copy / modify. There's already copious documentation, so I do not believe not having docs for Harry was ever an issue, since there have been plenty. 
>>> 
>>> You all are aware of my dedication to testing and quality of Apache Cassandra, and I hope you also see the benefits of having a model checker in-tree.
>>> 
>>> Thank you and happy upcoming holidays,
>>> --Alex
>>> 
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Joseph Lynch <jo...@gmail.com>.
+1

Sounds like a great change that will help us unify around a common testing
paradigm, and even pave the path to in-tree load testing plus integrated
correctness checking which would be extremely valuable!

-Joey

On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <ca...@gmail.com>
wrote:

> +1
>
> Agree w/ all the justifications mentioned above.
>
> As a reviewer on CASSANDRA-19210
> <https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were to
> a.) look at the directory, naming, and package structure of the ported
> code, b.) make sure IDE integration was working, and c.) make sure any
> modifications to existing code (rather than direct code movements from
> cassandra-harry) were straightforward.
>
> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:
>
>> Hey folks,
>>
>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>> trigger one more CI run overnight, and my intention was to merge it some
>> time soon, but I wanted to give a fair warning here, since this is a
>> relatively large patch.
>>
>> Good news for everyone that it:
>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>> namely) code that was using Harry already.
>>   b) the only tests that are changed are ones that used a duplicate
>> version of placement simulator we had both for testing TCM, and in Harry
>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>> have some base for examples/usage.
>>
>> Since we were effectively relying on this code for a while now, and the
>> intention now is to converge to:
>>   a) fewer different generators, and have a shareable version of
>> generators for everyone to use accross the base
>>   b) a testing tool that can be useful for both trivial cases, and
>> complex scenarios
>> myself and many other Cassandra contributors have expressed an opinion
>> that bringing Harry in-tree will be highly benefitial.
>>
>> I strongly believe that bringing Harry in-tree will help to lower the
>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>> Previously, it has been rather difficult to debug edge cases because I had
>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>> time consuming. Moreover, I believe we have missed at very least one RT
>> regression [2] because Harry was not in-tree, as its tests would've caught
>> the issue even with the model that existed.
>>
>> For other recently found issues, I think having Harry in-tree would have
>> substantially lowered a turnaround time, and allowed me to share repros
>> with developers of corresponding features much quicker.
>>
>> I do expect a slight learning curve for Harry, but my intention is to
>> build a web of simple tests (worked on some of them yesterday after
>> conversation with David already), which can follow the in-jvm-dtest pattern
>> of find-similar-test / copy / modify. There's already copious
>> documentation, so I do not believe not having docs for Harry was ever an
>> issue, since there have been plenty.
>>
>> You all are aware of my dedication to testing and quality of Apache
>> Cassandra, and I hope you also see the benefits of having a model checker
>> in-tree.
>>
>> Thank you and happy upcoming holidays,
>> --Alex
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>
>>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Caleb Rackliffe <ca...@gmail.com>.
+1

Agree w/ all the justifications mentioned above.

As a reviewer on CASSANDRA-19210
<https://issues.apache.org/jira/browse/CASSANDRA-19210>, my goals were to
a.) look at the directory, naming, and package structure of the ported
code, b.) make sure IDE integration was working, and c.) make sure any
modifications to existing code (rather than direct code movements from
cassandra-harry) were straightforward.

On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov <al...@coffeenco.de> wrote:

> Hey folks,
>
> I am mostly done with a patch that brings Harry in-tree [1]. I will
> trigger one more CI run overnight, and my intention was to merge it some
> time soon, but I wanted to give a fair warning here, since this is a
> relatively large patch.
>
> Good news for everyone that it:
>   a) touches no production code whatsoever. Only test (in-jvm dtest
> namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate
> version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to
> have some base for examples/usage.
>
> Since we were effectively relying on this code for a while now, and the
> intention now is to converge to:
>   a) fewer different generators, and have a shareable version of
> generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex
> scenarios
> myself and many other Cassandra contributors have expressed an opinion
> that bringing Harry in-tree will be highly benefitial.
>
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>
> I do expect a slight learning curve for Harry, but my intention is to
> build a web of simple tests (worked on some of them yesterday after
> conversation with David already), which can follow the in-jvm-dtest pattern
> of find-similar-test / copy / modify. There's already copious
> documentation, so I do not believe not having docs for Harry was ever an
> issue, since there have been plenty.
>
> You all are aware of my dedication to testing and quality of Apache
> Cassandra, and I hope you also see the benefits of having a model checker
> in-tree.
>
> Thank you and happy upcoming holidays,
> --Alex
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>
>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Ariel Weisberg <ar...@weisberg.ws>.
🥳🎉

Thanks for your work on this. Excited to have an easier way to write tests that leverage schema and data that also covers more.

Ariel
On Sat, Dec 23, 2023, at 9:17 AM, Alex Petrov wrote:
> Thanks everyone, Harry is now in tree! Of course, that's just a small milestone, hope it'll prove as useful as I expect it to be.
> 
> https://github.com/apache/cassandra/commit/439d1b122af334bf68c159b82ef4e4879c210bd5
> 
> Happy holidays!
> --Alex
> 
> On Sat, Dec 23, 2023, at 11:10 AM, Mick Semb Wever wrote:
>>    
>>   
>>> I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
>>> 
>>> For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
>> 
>> 
>> Agree, looking forward to getting to know and writing Harry tests.  Thank you Alex, happy holidays :) 
>> 
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Alex Petrov <al...@coffeenco.de>.
Thanks everyone, Harry is now in tree! Of course, that's just a small milestone, hope it'll prove as useful as I expect it to be.

https://github.com/apache/cassandra/commit/439d1b122af334bf68c159b82ef4e4879c210bd5

Happy holidays!
--Alex

On Sat, Dec 23, 2023, at 11:10 AM, Mick Semb Wever wrote:
>    
>   
>> I strongly believe that bringing Harry in-tree will help to lower the barrier for fuzz test and simplify co-development of Cassandra and Harry. Previously, it has been rather difficult to debug edge cases because I had to either re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar and bring it to Cassandra, which is both tedious and time consuming. Moreover, I believe we have missed at very least one RT regression [2] because Harry was not in-tree, as its tests would've caught the issue even with the model that existed.
>> 
>> For other recently found issues, I think having Harry in-tree would have substantially lowered a turnaround time, and allowed me to share repros with developers of corresponding features much quicker.
> 
> 
> Agree, looking forward to getting to know and writing Harry tests.  Thank you Alex, happy holidays :) 
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

Posted by Mick Semb Wever <mc...@apache.org>.
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>


Agree, looking forward to getting to know and writing Harry tests.  Thank
you Alex, happy holidays :)