You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Yuvraaj Kelkar <yu...@gmail.com> on 2023/01/26 06:11:37 UTC
RE%3A Re%3A Running all tests via Crave.io&In-Reply-To=
Apologies: I just signed up for this email list right now. I was signed up for the old Lucene list but didn't realize that the split would impact the mailing list as well. Explains a lot, now that I think about it.
I'll answer all the questions here, that I see from the thread:
1: (Noble) Full test for every PR raised: Ideally yes. The goal is to make sure that we can provide easy access to the sort of machines that can be used to really stress test Solr. Providing such "monster machines" only once in a while (eg once a day because of a crontab) goes against the whole idea of giving everyone enough infrastructure to have confidence in their own PR.
2: (Gus) Is Crave really prepared for anyone to come and use them: I'm going to just say "I sure hope so": We've been able to handle real life concurrent builds from 200+ developers. We've done synthetic testing to simulate thousands (4000) of developers and builds.
Also, if the question was: "Can Crave handle the cost of this infrastructure?": The answer is: Within reason, yes.
We try to reduce the cost of infrastructure by being clever about turning it on when needed and turning it off when idle. In other words, the duty cycle determines cost. There are also limits we've defined to ensure that the cost to Crave does not exceed a monthly budget.
I'm open to suggestions or funding.
3. (Gus) Is there a limit per person or project: Not really; do you want us to put limits?
4. (Gus) Does it require paid / membership:
Require: No.
There exists a paid version, but it is not required for open source projects.
5: (Ishaan) Can multiple developers execute their tests at the same time:
Yes.
Thanks,
-UV
On 2023/01/25 23:36:45 Noble Paul wrote:
> This is interesting.
>
> So, if the PR is merged , we will have the full test running on crave.io
> for every PR raised?
>
> On Thu, Jan 26, 2023 at 9:22 AM David Smiley wrote:
>
> > We haven't been running all our tests in GitHub Actions (i.e. PR
> > validation) because it was too time consuming to do so. I don't recall how
> > slow it was when someone last tried; it's probably better now but still
> > slow. To make up for this, there is a GHA only for SolrJ if a PR touches
> > SolrJ.
> >
> > There's now a PR here to introduce a new GHA that builds on Crave.io on a
> > beefy machine: https://github.com/apache/solr/pull/1303 The PR validation
> > took 11 minutes which is similar to the amount of time it took a GHA to
> > just do precommit checks -- 10 minutes :-)
> > I think we can remove the SolrJ specific GHA as it'll be redundant.
> >
> > Furthermore, anyone can use this to run tests from the convenience of your
> > laptop at the CLI while you are in the middle of any change (doesn't matter
> > what you have committed or not, pushed or not). To do so, run: crave run
> > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > -Ptests.jvms=48 test'
> >
> > Yeah that's long. There is a discussion in JIRA underway that may lead to
> > eliminating the "localSettings" step if, for example, it's moved to a bash
> > script executed by the gradle wrapper (my proposal). I should also be able
> > to configure crave with a default run configuration with this baked in.
> > I'll post an update when I'm able to do that.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: RE%3A Re%3A Running all tests via Crave.io&In-Reply-To=
Posted by Yuvraaj Kelkar <yu...@gmail.com>.
Yes, the stdout and stderr are streamed back to the initiator - the Github runner - which then records it as part of the Actions' logs.
On Jan 26 2023, at 5:11 am, Eric Pugh <ep...@opensourceconnections.com> wrote:
> Running the tests on every PR raised will make reviewing patches MUCH simpler. Today, on my laptop, it’s 40 minutes to 1 hour to run the full process, and I can’t use my machine during that time.. Being able to look at a PR and see that the tests run will make me more likely to review ;-).
>
> Does the output from Crave get integrated into Github Actions as a “Check”??
>
>
> > On Jan 26, 2023, at 1:11 AM, Yuvraaj Kelkar <yu...@gmail.com> wrote:
> >
> > Apologies: I just signed up for this email list right now. I was signed up for the old Lucene list but didn't realize that the split would impact the mailing list as well. Explains a lot, now that I think about it.
> >
> > I'll answer all the questions here, that I see from the thread:
> > 1: (Noble) Full test for every PR raised: Ideally yes. The goal is to make sure that we can provide easy access to the sort of machines that can be used to really stress test Solr. Providing such "monster machines" only once in a while (eg once a day because of a crontab) goes against the whole idea of giving everyone enough infrastructure to have confidence in their own PR.
> > 2: (Gus) Is Crave really prepared for anyone to come and use them: I'm going to just say "I sure hope so": We've been able to handle real life concurrent builds from 200+ developers. We've done synthetic testing to simulate thousands (4000) of developers and builds.
> > Also, if the question was: "Can Crave handle the cost of this infrastructure?": The answer is: Within reason, yes.
> > We try to reduce the cost of infrastructure by being clever about turning it on when needed and turning it off when idle. In other words, the duty cycle determines cost. There are also limits we've defined to ensure that the cost to Crave does not exceed a monthly budget.
> > I'm open to suggestions or funding.
> >
> > 3. (Gus) Is there a limit per person or project: Not really; do you want us to put limits?
> > 4. (Gus) Does it require paid / membership:
> > Require: No.
> > There exists a paid version, but it is not required for open source projects.
> >
> > 5: (Ishaan) Can multiple developers execute their tests at the same time:
> > Yes.
> >
> > Thanks,
> > -UV
> >
> > On 2023/01/25 23:36:45 Noble Paul wrote:
> >> This is interesting.
> >>
> >> So, if the PR is merged , we will have the full test running on crave.io
> >> for every PR raised?
> >>
> >> On Thu, Jan 26, 2023 at 9:22 AM David Smiley wrote:
> >>
> >>> We haven't been running all our tests in GitHub Actions (i.e. PR
> >>> validation) because it was too time consuming to do so. I don't recall how
> >>> slow it was when someone last tried; it's probably better now but still
> >>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches
> >>> SolrJ.
> >>>
> >>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
> >>> beefy machine: https://github.com/apache/solr/pull/1303 The PR validation
> >>> took 11 minutes which is similar to the amount of time it took a GHA to
> >>> just do precommit checks -- 10 minutes :-)
> >>> I think we can remove the SolrJ specific GHA as it'll be redundant.
> >>>
> >>> Furthermore, anyone can use this to run tests from the convenience of your
> >>> laptop at the CLI while you are in the middle of any change (doesn't matter
> >>> what you have committed or not, pushed or not). To do so, run: crave run
> >>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> >>> -Ptests.jvms=48 test'
> >>>
> >>> Yeah that's long. There is a discussion in JIRA underway that may lead to
> >>> eliminating the "localSettings" step if, for example, it's moved to a bash
> >>> script executed by the gradle wrapper (my proposal). I should also be able
> >>> to configure crave with a default run configuration with this baked in.
> >>> I'll post an update when I'm able to do that.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
Re: RE%3A Re%3A Running all tests via Crave.io&In-Reply-To=
Posted by Jan Høydahl <ja...@cominvent.com>.
I like this.
Perhaps it will also put more focus on fixing bad tests, since some PRs are bound to fail the full test suite due to test flakiness, but it will be more evident. Which is a good thing.
Jan
> 26. jan. 2023 kl. 12:11 skrev Eric Pugh <ep...@opensourceconnections.com>:
>
> Running the tests on every PR raised will make reviewing patches MUCH simpler. Today, on my laptop, it’s 40 minutes to 1 hour to run the full process, and I can’t use my machine during that time.. Being able to look at a PR and see that the tests run will make me more likely to review ;-).
>
> Does the output from Crave get integrated into Github Actions as a “Check”??
>
>
>
>> On Jan 26, 2023, at 1:11 AM, Yuvraaj Kelkar <yu...@gmail.com> wrote:
>>
>> Apologies: I just signed up for this email list right now. I was signed up for the old Lucene list but didn't realize that the split would impact the mailing list as well. Explains a lot, now that I think about it.
>>
>> I'll answer all the questions here, that I see from the thread:
>> 1: (Noble) Full test for every PR raised: Ideally yes. The goal is to make sure that we can provide easy access to the sort of machines that can be used to really stress test Solr. Providing such "monster machines" only once in a while (eg once a day because of a crontab) goes against the whole idea of giving everyone enough infrastructure to have confidence in their own PR.
>> 2: (Gus) Is Crave really prepared for anyone to come and use them: I'm going to just say "I sure hope so": We've been able to handle real life concurrent builds from 200+ developers. We've done synthetic testing to simulate thousands (4000) of developers and builds.
>> Also, if the question was: "Can Crave handle the cost of this infrastructure?": The answer is: Within reason, yes.
>> We try to reduce the cost of infrastructure by being clever about turning it on when needed and turning it off when idle. In other words, the duty cycle determines cost. There are also limits we've defined to ensure that the cost to Crave does not exceed a monthly budget.
>> I'm open to suggestions or funding.
>>
>> 3. (Gus) Is there a limit per person or project: Not really; do you want us to put limits?
>> 4. (Gus) Does it require paid / membership:
>> Require: No.
>> There exists a paid version, but it is not required for open source projects.
>>
>> 5: (Ishaan) Can multiple developers execute their tests at the same time:
>> Yes.
>>
>> Thanks,
>> -UV
>>
>> On 2023/01/25 23:36:45 Noble Paul wrote:
>>> This is interesting.
>>>
>>> So, if the PR is merged , we will have the full test running on crave.io
>>> for every PR raised?
>>>
>>> On Thu, Jan 26, 2023 at 9:22 AM David Smiley wrote:
>>>
>>>> We haven't been running all our tests in GitHub Actions (i.e. PR
>>>> validation) because it was too time consuming to do so. I don't recall how
>>>> slow it was when someone last tried; it's probably better now but still
>>>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches
>>>> SolrJ.
>>>>
>>>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
>>>> beefy machine: https://github.com/apache/solr/pull/1303 The PR validation
>>>> took 11 minutes which is similar to the amount of time it took a GHA to
>>>> just do precommit checks -- 10 minutes :-)
>>>> I think we can remove the SolrJ specific GHA as it'll be redundant.
>>>>
>>>> Furthermore, anyone can use this to run tests from the convenience of your
>>>> laptop at the CLI while you are in the middle of any change (doesn't matter
>>>> what you have committed or not, pushed or not). To do so, run: crave run
>>>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
>>>> -Ptests.jvms=48 test'
>>>>
>>>> Yeah that's long. There is a discussion in JIRA underway that may lead to
>>>> eliminating the "localSettings" step if, for example, it's moved to a bash
>>>> script executed by the gradle wrapper (my proposal). I should also be able
>>>> to configure crave with a default run configuration with this baked in.
>>>> I'll post an update when I'm able to do that.
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: RE%3A Re%3A Running all tests via Crave.io&In-Reply-To=
Posted by Eric Pugh <ep...@opensourceconnections.com>.
Running the tests on every PR raised will make reviewing patches MUCH simpler. Today, on my laptop, it’s 40 minutes to 1 hour to run the full process, and I can’t use my machine during that time.. Being able to look at a PR and see that the tests run will make me more likely to review ;-).
Does the output from Crave get integrated into Github Actions as a “Check”??
> On Jan 26, 2023, at 1:11 AM, Yuvraaj Kelkar <yu...@gmail.com> wrote:
>
> Apologies: I just signed up for this email list right now. I was signed up for the old Lucene list but didn't realize that the split would impact the mailing list as well. Explains a lot, now that I think about it.
>
> I'll answer all the questions here, that I see from the thread:
> 1: (Noble) Full test for every PR raised: Ideally yes. The goal is to make sure that we can provide easy access to the sort of machines that can be used to really stress test Solr. Providing such "monster machines" only once in a while (eg once a day because of a crontab) goes against the whole idea of giving everyone enough infrastructure to have confidence in their own PR.
> 2: (Gus) Is Crave really prepared for anyone to come and use them: I'm going to just say "I sure hope so": We've been able to handle real life concurrent builds from 200+ developers. We've done synthetic testing to simulate thousands (4000) of developers and builds.
> Also, if the question was: "Can Crave handle the cost of this infrastructure?": The answer is: Within reason, yes.
> We try to reduce the cost of infrastructure by being clever about turning it on when needed and turning it off when idle. In other words, the duty cycle determines cost. There are also limits we've defined to ensure that the cost to Crave does not exceed a monthly budget.
> I'm open to suggestions or funding.
>
> 3. (Gus) Is there a limit per person or project: Not really; do you want us to put limits?
> 4. (Gus) Does it require paid / membership:
> Require: No.
> There exists a paid version, but it is not required for open source projects.
>
> 5: (Ishaan) Can multiple developers execute their tests at the same time:
> Yes.
>
> Thanks,
> -UV
>
> On 2023/01/25 23:36:45 Noble Paul wrote:
>> This is interesting.
>>
>> So, if the PR is merged , we will have the full test running on crave.io
>> for every PR raised?
>>
>> On Thu, Jan 26, 2023 at 9:22 AM David Smiley wrote:
>>
>>> We haven't been running all our tests in GitHub Actions (i.e. PR
>>> validation) because it was too time consuming to do so. I don't recall how
>>> slow it was when someone last tried; it's probably better now but still
>>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches
>>> SolrJ.
>>>
>>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
>>> beefy machine: https://github.com/apache/solr/pull/1303 The PR validation
>>> took 11 minutes which is similar to the amount of time it took a GHA to
>>> just do precommit checks -- 10 minutes :-)
>>> I think we can remove the SolrJ specific GHA as it'll be redundant.
>>>
>>> Furthermore, anyone can use this to run tests from the convenience of your
>>> laptop at the CLI while you are in the middle of any change (doesn't matter
>>> what you have committed or not, pushed or not). To do so, run: crave run
>>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
>>> -Ptests.jvms=48 test'
>>>
>>> Yeah that's long. There is a discussion in JIRA underway that may lead to
>>> eliminating the "localSettings" step if, for example, it's moved to a bash
>>> script executed by the gradle wrapper (my proposal). I should also be able
>>> to configure crave with a default run configuration with this baked in.
>>> I'll post an update when I'm able to do that.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.