Re: [PR] chore: trim CI builds to one full chain per module and lane [hudi]

["wombatu-kun (via GitHub)" <gi...@apache.org>]

wombatu-kun commented on code in PR #19909:
URL: https://github.com/apache/hudi/pull/19909#discussion_r4346703087


##########
.github/workflows/bot.yml:
##########
@@ -273,26 +259,34 @@ jobs:
 
     env:
       UT_MODULES: >-
-        !hudi-hadoop-common,!hudi-hadoop-mr,!hudi-client/hudi-java-client,!hudi-client/hudi-spark-client,!hudi-cli,!hudi-examples,!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-spark,!hudi-spark-datasource,!hudi-spark-datasource/hudi-spark,!hudi-spark-datasource/hudi-spark3.5.x,!hudi-spark-datasource/hudi-spark3-common,!hudi-spark-datasource/hudi-spark-common,!hudi-utilities
+        !hudi-hadoop-common,!hudi-hadoop-mr,!hudi-client/hudi-java-client,!hudi-client/hudi-spark-client,!hudi-cli,!hudi-examples,!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-k8s,!hudi-examples/hudi-examples-spark,!hudi-spark-datasource,!hudi-spark-datasource/hudi-spark,!hudi-spark-datasource/hudi-spark3.5.x,!hudi-spark-datasource/hudi-spark3-common,!hudi-spark-datasource/hudi-spark-common,!hudi-utilities
       FT_MODULES: >-
-        !hudi-client/hudi-spark-client,!hudi-cli,!hudi-examples,!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-spark,!hudi-spark-datasource/hudi-spark,!hudi-utilities
+        !hudi-client/hudi-spark-client,!hudi-cli,!hudi-examples,!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-k8s,!hudi-examples/hudi-examples-spark,!hudi-spark-datasource/hudi-spark,!hudi-utilities
+      # The fourteen bundles of the default reactor. None has tests, validate-bundles shades and
+      # exercises every one, and nothing this job tests depends on them, so the build and the two
+      # test steps below all leave them out (a bundle left in a test reactor would try to resolve
+      # the modules the trimmed build no longer installs).
+      BUNDLE_MODULES: >-

Review Comment:
   #19674 adds `packaging/hudi-spark-native-bundle` to the default reactor with a compile dependency on the spark bundle, so once both land this job's build fails to resolve the spark bundle unless the new bundle joins this list. Could the comment above say every new bundle has to be listed here, so whichever PR lands second knows where to look?



##########
.github/workflows/java_ci_engines.yml:
##########
@@ -0,0 +1,617 @@
+name: Java CI Engines
+
+# This filter replaces the former `changes` job. That job listed the changed files and every test
+# job waited on it through `needs`, which made each run wait for a shared runner twice: once for
+# the filter, once for the tests. GitHub evaluates `paths` before creating any job, so the same
+# decision now costs nothing. A docs-only change creates no run of this workflow at all, which is
+# fine because no required status check lives here (see validate_source.yml).
+# yml/yaml files are deliberately NOT ignored, unlike the old filter: docker/compose/*.yml drives
+# the integration tests, several modules keep yml test resources, and the workflow files themselves
+# need CI to run when they change.
+on:
+  push:
+    branches:
+      - master
+      - 'release-*'
+      - branch-0.x
+    paths:
+      - '**'
+      # Listed explicitly so the filter does not depend on '**' matching dot-prefixed paths.
+      - '.github/**'
+      - '!**.bmp'
+      - '!**.gif'
+      - '!**.jpg'
+      - '!**.jpeg'
+      - '!**.md'
+      - '!**.pdf'
+      - '!**.png'
+      - '!**.svg'
+      - '!.gitignore'
+  pull_request:
+    branches:
+      - master
+      - 'release-*'
+      - branch-0.x
+    paths:
+      - '**'
+      # Listed explicitly so the filter does not depend on '**' matching dot-prefixed paths.
+      - '.github/**'
+      - '!**.bmp'
+      - '!**.gif'
+      - '!**.jpg'
+      - '!**.jpeg'
+      - '!**.md'
+      - '!**.pdf'
+      - '!**.png'
+      - '!**.svg'
+      - '!.gitignore'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ !contains(github.ref, 'master') && !contains(github.ref, 'branch-0.x') && !contains(github.ref, 'release-') }}
+
+env:
+  # The retry/timeout knobs below are maven-resolver (aether.*) properties. Maven 3.9 resolves
+  # artifacts through the native resolver HTTP transport, which ignores the older maven.wagon.* names.
+  # serviceUnavailable lists the HTTP codes retried instead of failing the build. 403 is deliberately
+  # absent: Maven Central answers 403 when its CDN has blocked the runner's egress IP, and retrying
+  # that is what escalates the block.
+  # Keep MVN_ARGS in sync with bot.yml.
+  MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn -Daether.connector.http.retryHandler.count=5 -Daether.connector.http.retryHandler.interval=3000 -Daether.connector.http.retryHandler.intervalMax=30000 -Daether.connector.http.retryHandler.serviceUnavailable=429,500,502,503,504 -Daether.connector.http.connectionMaxTtl=25 -Dorg.eclipse.jetty.LEVEL=WARN
+  FLINK_IT_FILTER1: -Dit.test=ITTestHoodieDataSource
+  # An include is required next to the exclusion: a bare exclusion replaces failsafe's IT*
+  # include pattern with every test class, which ran hudi-flink's unit tests a second time here.
+  FLINK_IT_FILTER2: -Dit.test=ITTest*,!ITTestHoodieDataSource

Review Comment:
   `ITTest*` is narrower than the `**/IT*.java` include this filter replaces, so a future hudi-flink IT not named `ITTest*` would run in no job. Could it be `IT*,!ITTestHoodieDataSource`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org