You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2025/06/23 10:25:49 UTC
[PR] HDDS-13322. flaky-test-check cannot detect module for some tests [ozone]
adoroszlai opened a new pull request, #8679:
URL: https://github.com/apache/ozone/pull/8679
## What changes were proposed in this pull request?
Remove automatic module detection from `flaky-test-check`.
1. It cannot detect some test classes, e.g. `TestOzoneIntegrationNonHA`. This is due to having to use old Surefire plugin, because new version cannot reliably kill the fork.
2. With undetected module it has to build all modules. `hdds-docs` runs `docs.sh` as part of the build, which truncates `output.log`. This causes only the last iteration's result being kept in `OUTPUT_DIR`.
3. Manually setting the submodule is faster in all cases.
Thus I think it's best to remove `AUTO` as an option and make this input mandatory.
https://issues.apache.org/jira/browse/HDDS-13322
## How was this patch tested?
https://github.com/adoroszlai/ozone/actions/runs/15821433077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8679:
URL: https://github.com/apache/ozone/pull/8679#discussion_r2161793475
##########
.github/workflows/intermittent-test-check.yml:
##########
@@ -95,54 +94,11 @@ jobs:
with:
repo: ${{ github.event.inputs.ratis-repo || format('{0}/ratis', github.repository_owner) }}
ref: ${{ github.event.inputs.ratis-ref }}
- find-tests:
- if: ${{ always() && github.event.inputs.submodule == 'AUTO' }}
- needs:
- - prepare-job
- runs-on: ubuntu-24.04
- outputs:
- modules: ${{ steps.modules.outputs.modules }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ github.event.inputs.ref }}
- - name: Cache for maven dependencies
- uses: actions/cache/restore@v4
- with:
- path: |
- ~/.m2/repository/*/*/*
- !~/.m2/repository/org/apache/ozone
- key: maven-repo-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-repo-
- - name: Setup java
- uses: actions/setup-java@v4
- with:
- distribution: 'temurin'
- java-version: ${{ github.event.inputs.java-version }}
- - name: Find tests
- run: |
- # find tests to be run in splits by running them with very short timeout
- hadoop-ozone/dev-support/checks/junit.sh -DexcludedGroups="native|slow|unhealthy" -DskipShade \
- -Dtest="$TEST_CLASS,Abstract*Test*\$*" \
- -Dsurefire.fork.timeout=1 -Dmaven-surefire-plugin.version=${{ env.SUREFIRE_VERSION }} \
Review Comment:
We use new Surefire for actual test execution, only module detection needs the old one, to be able to kill the fork. New versions may run "forever" (until workflow job timeout, [example](https://github.com/apache/ozone/actions/runs/15794201498/job/44524376589#step:13:4777)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8679:
URL: https://github.com/apache/ozone/pull/8679#discussion_r2161940890
##########
.github/workflows/intermittent-test-check.yml:
##########
@@ -95,54 +94,11 @@ jobs:
with:
repo: ${{ github.event.inputs.ratis-repo || format('{0}/ratis', github.repository_owner) }}
ref: ${{ github.event.inputs.ratis-ref }}
- find-tests:
- if: ${{ always() && github.event.inputs.submodule == 'AUTO' }}
- needs:
- - prepare-job
- runs-on: ubuntu-24.04
- outputs:
- modules: ${{ steps.modules.outputs.modules }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ github.event.inputs.ref }}
- - name: Cache for maven dependencies
- uses: actions/cache/restore@v4
- with:
- path: |
- ~/.m2/repository/*/*/*
- !~/.m2/repository/org/apache/ozone
- key: maven-repo-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-repo-
- - name: Setup java
- uses: actions/setup-java@v4
- with:
- distribution: 'temurin'
- java-version: ${{ github.event.inputs.java-version }}
- - name: Find tests
- run: |
- # find tests to be run in splits by running them with very short timeout
- hadoop-ozone/dev-support/checks/junit.sh -DexcludedGroups="native|slow|unhealthy" -DskipShade \
- -Dtest="$TEST_CLASS,Abstract*Test*\$*" \
- -Dsurefire.fork.timeout=1 -Dmaven-surefire-plugin.version=${{ env.SUREFIRE_VERSION }} \
Review Comment:
> So IIUC we cannot use new version because of the fork not getting terminated correctly and can't use the old version as it does not print the module correctly for some reason?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8679:
URL: https://github.com/apache/ozone/pull/8679#issuecomment-3000616656
Thanks @sadanand48 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #8679:
URL: https://github.com/apache/ozone/pull/8679#discussion_r2161904691
##########
.github/workflows/intermittent-test-check.yml:
##########
@@ -95,54 +94,11 @@ jobs:
with:
repo: ${{ github.event.inputs.ratis-repo || format('{0}/ratis', github.repository_owner) }}
ref: ${{ github.event.inputs.ratis-ref }}
- find-tests:
- if: ${{ always() && github.event.inputs.submodule == 'AUTO' }}
- needs:
- - prepare-job
- runs-on: ubuntu-24.04
- outputs:
- modules: ${{ steps.modules.outputs.modules }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ github.event.inputs.ref }}
- - name: Cache for maven dependencies
- uses: actions/cache/restore@v4
- with:
- path: |
- ~/.m2/repository/*/*/*
- !~/.m2/repository/org/apache/ozone
- key: maven-repo-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-repo-
- - name: Setup java
- uses: actions/setup-java@v4
- with:
- distribution: 'temurin'
- java-version: ${{ github.event.inputs.java-version }}
- - name: Find tests
- run: |
- # find tests to be run in splits by running them with very short timeout
- hadoop-ozone/dev-support/checks/junit.sh -DexcludedGroups="native|slow|unhealthy" -DskipShade \
- -Dtest="$TEST_CLASS,Abstract*Test*\$*" \
- -Dsurefire.fork.timeout=1 -Dmaven-surefire-plugin.version=${{ env.SUREFIRE_VERSION }} \
Review Comment:
So IIUC we cannot use new version because of the fork not getting terminated correctly and can't use the old version as it does not print the module correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #8679:
URL: https://github.com/apache/ozone/pull/8679#discussion_r2161426818
##########
.github/workflows/intermittent-test-check.yml:
##########
@@ -95,54 +94,11 @@ jobs:
with:
repo: ${{ github.event.inputs.ratis-repo || format('{0}/ratis', github.repository_owner) }}
ref: ${{ github.event.inputs.ratis-ref }}
- find-tests:
- if: ${{ always() && github.event.inputs.submodule == 'AUTO' }}
- needs:
- - prepare-job
- runs-on: ubuntu-24.04
- outputs:
- modules: ${{ steps.modules.outputs.modules }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ github.event.inputs.ref }}
- - name: Cache for maven dependencies
- uses: actions/cache/restore@v4
- with:
- path: |
- ~/.m2/repository/*/*/*
- !~/.m2/repository/org/apache/ozone
- key: maven-repo-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-repo-
- - name: Setup java
- uses: actions/setup-java@v4
- with:
- distribution: 'temurin'
- java-version: ${{ github.event.inputs.java-version }}
- - name: Find tests
- run: |
- # find tests to be run in splits by running them with very short timeout
- hadoop-ozone/dev-support/checks/junit.sh -DexcludedGroups="native|slow|unhealthy" -DskipShade \
- -Dtest="$TEST_CLASS,Abstract*Test*\$*" \
- -Dsurefire.fork.timeout=1 -Dmaven-surefire-plugin.version=${{ env.SUREFIRE_VERSION }} \
Review Comment:
If the issue is due to surefire version , we could just set to it to one where it works only for the module detection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #8679:
URL: https://github.com/apache/ozone/pull/8679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13322. Remove module auto-detection from flaky-test-check [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #8679:
URL: https://github.com/apache/ozone/pull/8679#discussion_r2161904691
##########
.github/workflows/intermittent-test-check.yml:
##########
@@ -95,54 +94,11 @@ jobs:
with:
repo: ${{ github.event.inputs.ratis-repo || format('{0}/ratis', github.repository_owner) }}
ref: ${{ github.event.inputs.ratis-ref }}
- find-tests:
- if: ${{ always() && github.event.inputs.submodule == 'AUTO' }}
- needs:
- - prepare-job
- runs-on: ubuntu-24.04
- outputs:
- modules: ${{ steps.modules.outputs.modules }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ github.event.inputs.ref }}
- - name: Cache for maven dependencies
- uses: actions/cache/restore@v4
- with:
- path: |
- ~/.m2/repository/*/*/*
- !~/.m2/repository/org/apache/ozone
- key: maven-repo-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-repo-
- - name: Setup java
- uses: actions/setup-java@v4
- with:
- distribution: 'temurin'
- java-version: ${{ github.event.inputs.java-version }}
- - name: Find tests
- run: |
- # find tests to be run in splits by running them with very short timeout
- hadoop-ozone/dev-support/checks/junit.sh -DexcludedGroups="native|slow|unhealthy" -DskipShade \
- -Dtest="$TEST_CLASS,Abstract*Test*\$*" \
- -Dsurefire.fork.timeout=1 -Dmaven-surefire-plugin.version=${{ env.SUREFIRE_VERSION }} \
Review Comment:
So IIUC we cannot use new version because of the fork not getting terminated correctly and can't use the old version as it does not print the module correctly for some reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org