You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2023/01/18 23:25:14 UTC
[GitHub] [solr] uvatbc opened a new pull request, #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
uvatbc opened a new pull request, #1303:
URL: https://github.com/apache/solr/pull/1303
https://issues.apache.org/jira/browse/SOLR-XXXXX
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
The Solr project needs machines that are much more powerful (eg. 100+ cores) to run all the tests.
Crave.io offers a simple and cost effective way to provision such machines
# Solution
1. Create a separate yaml so that its possible to test the new functionality without impacting current workflows
1. Add a stage: "Download crave binary"
1. Prefix `crave run` behind the two gradlew commands
This should utlilize the "Zeroconf" Crave set up.
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Have not tested because it is a pipeline change that requires pipeline access.
# Checklist
Please review the following and check all that apply:
- [X] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [X] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084774842
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
Review Comment:
Agreed. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084776393
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
Re first comment: Yes, the output from `localSettings` will be retained between subsequent `crave run`s.
Re second comment: I've changed the worker count to match the core count on the remote build/test node so that it matches whatever the ephemeral node is configured to be.
In other words, when we use a 16 core machine, we'll use 16 workers. When we configure crave to use `n2d-standard-224` instances, `nproc` will return 224 and automatically tell gradlew to use the maximum number of cores.
We can consider a linear scale factor for the number of cores in a later change as we understand or tweak how well gradle performs on multiple cores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084829522
##########
.github/workflows/tests-via-crave.yml:
##########
@@ -0,0 +1,38 @@
+name: Solr Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+
+jobs:
+ test:
+ name: Run Solr Tests using Crave.io resources
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
Review Comment:
I suspect we don't need any of this because we won't be running the build at all on this GitHub Action machine.
##########
.github/workflows/tests-via-crave.yml:
##########
@@ -0,0 +1,38 @@
+name: Solr Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+
+jobs:
+ test:
+ name: Run Solr Tests using Crave.io resources
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
+ - name: Test the Solr Package
+ run: ./crave run -- './gradlew --max-workers=`nproc` test'
Review Comment:
```suggestion
run: ./crave run -- './gradlew --max-workers=`nproc` -Ptests.jvms=48 test'
```
I did some exploration and found that it's not enough to set the workers in this way as the "tests.jvms" will default to the value in gradle.properties which is again capped at 12. When I used 48 workers & test JVMs on this machine, the tests finished in 5:45 which is awesome! If you wish we could use nproc again thus 96 for both. I did one such build and it pegged the cpu but then stalled out, probably due to misbehavior in a test or something. I ultimately killed it. Could just be bad luck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1403146619
Looking at the GH Action results, the crave executable failed because of no crave conf:
> Error: Configuration file not found. Zero conf not found: No matching projects found for URL "https://github.com/apache/solr"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084774129
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
Review Comment:
Fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1403144847
I imagine Crave might work on a "self hosted runner" _eventually_ instead of indirectly using GitHub runner:
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084822103
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
Half the number of available processors reported by the JVM; the JVM counts hyper threads individually. Since 2x threads per core is common, our build divided this by two to guess at the number of physical cores. Then this is capped at 12. Logic is in `./gradle/generation/local-settings.gradle`. There are lots of concurrent threads in the JVM so I this this approach of using physical cores in spite of hyper threading makes sense. On a machine with 96 threads on 48 cores (which I used last weekend playing around on Crave.io), 48 gradle workers worked well (and so did 24).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1404227076
There's ways to eliminate both those "startup" times. Just to note them down:
Downloading Solr deps: That might be something to be put into the Docker container assigned to the Solr project. Can you point me to some logs to understand whats being downloaded? I'll try to put all of it into the container so that its not being pulled repeatedly.
Fetch all the branches: We may have a trick for that that involves eliminating the clone altogether, but let me try it out myself before I confirm it works.
On Jan 25 2023, at 3:04 pm, David Smiley ***@***.***> wrote:
>
>
> If I'm not mistaken, there are 96 hyper threads on 48 cores?
> The first two minutes in Crave was Crave downloading Solr's dependencies :-(. Hopefully won't repeat for each build.
> Also the "fetch all branches" option I used was a blunt approach to satisfy what Crave needs. There may be a better way. A minute was lost in checking everything out -- dunno if it'll be repeated. I'll re-run this to see.
>
>
> The net of these delays and actual test execution was 11 minutes (as we can see here in the "all checks have passed" summary. I'm happy with that :-)
> —
> Reply to this email directly, view it on GitHub ***@***.***/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fsolr%2Fpull%2F1303%23issuecomment-1404220683&recipient=cmVwbHkrQUNBQVVPSlhBVjZTRVdSRFJBTkVQQkdCMzNGNEpFVkJOSEhGWVNGNk1JQHJlcGx5LmdpdGh1Yi5jb20%3D), or unsubscribe ***@***.***/1?redirect=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACAAUOPPPV5YH4BUMPXZMC3WUGIMJANCNFSM6AAAAAAT7VV2TA&recipient=cmVwbHkrQUNBQVVPSlhBVjZTRVdSRFJBTkVQQkdCMzNGNEpFVkJOSEhGWVNGNk1JQHJlcGx5LmdpdGh1Yi5jb20%3D).
> You are receiving this because you authored the thread.
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1087107239
##########
.github/workflows/tests-via-crave.yml:
##########
@@ -12,8 +12,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- # Setup
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
Review Comment:
gets rid of the annoying "node needs updating" message too!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1403758318
Added the link "https://github.com/apache/solr" to the ZeroConf configuration as another alias for the Solr git repository.
The Github action should now work: The client binary should be able to recognize this as a ZeroConf request.
@Self hosted runner: Makes sense. Next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley merged pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley merged PR #1303:
URL: https://github.com/apache/solr/pull/1303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084773938
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
+ - name: Test the SolrJ Package
+ run: ./crave run -- ./gradlew test
Review Comment:
Renamed as requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084786595
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
@elyograg `nproc` seems to return "the number of cores available to your process". Through empirical testing on my X1 carbon, this seems to mean "hyperthreaded cores".
Testing on the `n1-standard-96` set up that we have for Solr testing got me:
```
19:23:12 uv@uv-x1:~/src/github.com/java/solr/solr$ crave ssh -- nproc
96
```
So it looks like it is hyperthreaded cores for sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1086751334
##########
.github/workflows/tests-via-crave.yml:
##########
@@ -0,0 +1,38 @@
+name: Solr Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+
+jobs:
+ test:
+ name: Run Solr Tests using Crave.io resources
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1404210784
Ok so that worked... Thanks @dsmiley !
Looks like really good CPU utilization of the 96 cores.
<img width="990" alt="image" src="https://user-images.githubusercontent.com/8391225/214688056-4082afe3-80b5-43b8-bb55-2434044b0edb.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084758269
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
+ - name: Test the SolrJ Package
+ run: ./crave run -- ./gradlew test
Review Comment:
The name of this file suggests the tests are specific to SolrJ yet this line here will run all tests. Lets rename this file to `tests-via-crave.yml` or something like that. After all, the point of this is that we'll be able to run all tests efficiently, not just one module's tests within Solr.
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
FYI this is going to write a gradle.properties file to the project dir if it isn't there; if it is there already, it'll do nothing. Will this file be retained between builds?
The code that generates this gradle.properties has a hard cap on the number of CPUs to use at 12. I could try adding options to the localSettings task to not cap it. If we are okay with hard-coding into this GitHub actions yaml script a particular number of test runners based on assuming knowledge of what type of machine is going to run it, it'd be an easy work-around. That would be `./gradlew --max-workers=48 test` below.
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
Review Comment:
```suggestion
name: Run Solr Tests using Crave.io resources
```
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
Review Comment:
```suggestion
name: Solr Tests
```
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
Review Comment:
```suggestion
```
Lets run all tests
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
There are other CLI options that may be useful as well like disabling the gradle daemon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084774519
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] elyograg commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "elyograg (via GitHub)" <gi...@apache.org>.
elyograg commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1084781451
##########
.github/workflows/solrj-test-crave.yml:
##########
@@ -0,0 +1,41 @@
+name: SolrJ Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - '.github/workflows/solrj-test.yml'
+ - 'solr/solrj/**'
+
+jobs:
+ test:
+ name: Run SolrJ Tests
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
Review Comment:
Does that return the number of REAL cpu cores, or the same count you would see in something like /proc/cpuinfo? I would wager that most modern CPUs support hyperthreading, which makes most programs think my 24 core server has 48 cores. I don't want to disable HT, because even though it's not a true doubling of the core count, it does help some workloads run faster. But I wouldn't want the solr build to decide it can handle 48 workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1404220683
If I'm not mistaken, there are 96 hyper threads on 48 *cores*?
The first two minutes in Crave was Crave downloading Solr's dependencies :-(. Hopefully won't repeat for each build.
Also the "fetch all branches" option I used was a blunt approach to satisfy what Crave needs. There may be a better way. A minute was lost in checking everything out -- dunno if it'll be repeated. I'll re-run this to see.
The net of these delays and actual test execution was 11 minutes (as we can see here in the "all checks have passed" summary. I'm happy with that :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #1303:
URL: https://github.com/apache/solr/pull/1303#issuecomment-1404252037
I think I'm done tweaking this thing. Still 11 minutes, although the milestones of what happens changed a bit. The checkout-everything delay is still there, as are Crave side delays of downloading everything. Nevertheless; this is awesome -- worth merging! I want to bring attention to this issue on the Solr dev list.
Crafting a custom Docker build image with the dependencies isn't worth the maintenance. Build infrastructure (be it GitHub Actions or Crave) needs a feature of caching certain directories to avoid repeated work. I saw this in GHA. A fast network-local caching proxy (either HTTP generic or Maven repo specific) could ameliorate that and be complementary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uvatbc commented on a diff in pull request #1303: Add temporary pipeline to test Solr using machines provided by Crave.io
Posted by "uvatbc (via GitHub)" <gi...@apache.org>.
uvatbc commented on code in PR #1303:
URL: https://github.com/apache/solr/pull/1303#discussion_r1086751963
##########
.github/workflows/tests-via-crave.yml:
##########
@@ -0,0 +1,38 @@
+name: Solr Tests
+
+on:
+ pull_request:
+ branches:
+ - 'main'
+
+jobs:
+ test:
+ name: Run Solr Tests using Crave.io resources
+
+ runs-on: ubuntu-latest
+
+ steps:
+ # Setup
+ - uses: actions/checkout@v2
+ - name: Set up JDK 11
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 11
+ java-package: jdk
+ - name: Grant execute permission for gradlew
+ run: chmod +x gradlew
+ - uses: actions/cache@v2
+ with:
+ path: |
+ ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-solrj-${{ hashFiles('versions.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-gradle-solrj-
+ ${{ runner.os }}-gradle-
+ - name: Get the Crave binary
+ run: curl -s https://raw.githubusercontent.com/accupara/crave/master/get_crave.sh | bash -s --
+ - name: Initialize gradle settings
+ run: ./crave run -- ./gradlew localSettings
+ - name: Test the Solr Package
+ run: ./crave run -- './gradlew --max-workers=`nproc` test'
Review Comment:
Updated as requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org