You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org> on 2026/05/15 06:43:06 UTC

[PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Aggarwal-Raghav opened a new pull request, #498:
URL: https://github.com/apache/tez/pull/498

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255188398


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}
+  done
+}
+
+## @description  File filter to determine which tests to run
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_file_filter
+{
+  local filename=$1
+
+  yetus_debug "Tez file filter: ${filename}"

Review Comment:
   I checked runs no.7 and no.8 and haven't seen "Tez file filter", am I supposed to see it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4495186384

   > @abstractdog , the PR is ready from my side. I think the improvement is good. Evidence, for #493 , compare the 2 runs lastet Jenkinsfile: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-500/3/Yetus_20Report/ Older Jenkinsfile: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-493/9/Yetus_20Report/
   > 
   > Metric	Older	Newer
   > **Time Taken**	242 min	166 min
   > **UT Execution Level**	Individual Submodule	Root (Project Level)
   > **`mvn install` Status**	**Failure** (Older submodule dependency snapshot issues)	**Success (Green)** (Build at project level)
   > **All in All we are adding approx 70 min of UT in every PR** . spotbugs, linter, javadoc time will be dependent on PR changes as they are retaining the behaviour of running on submodule level.
   
   it's fine, I'm glad we achieved full coverage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4495207433

   > Thanks, once https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/ completes we can get this merged 👍🏻
   
   oops, already merged...let's follow that run in case of any problems :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4462711417

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 41s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   9m 51s |  |  master passed  |
   | +1 :green_heart: |  compile  |   5m 16s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   7m 38s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   2m 37s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  compile  |   5m 43s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  2s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  95m 16s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/7/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 15s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 131m 18s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | tez.dag.app.rm.TestTaskScheduler |
   |   | tez.test.TestTezJobs |
   |   | tez.test.TestAMRecoveryAggregationBroadcast |
   |   | tez.runtime.library.conf.TestOrderedPartitionedKVEdgeConfig |
   |   | tez.dag.library.vertexmanager.TestShuffleVertexManager |
   |   | tez.runtime.library.conf.TestUnorderedPartitionedKVEdgeConfig |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/7/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 588021b46ebb 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / 383ffd991a4f67babf218b28c3b9193835c4fa25 |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/7/testReport/ |
   | Max. process+thread count | 1370 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/7/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4472097720

   Most of the tez-personality.sh was inspired from https://github.com/apache/phoenix/blob/master/dev/phoenix-personality.sh will address your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4472465286

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m 59s |  |  master passed  |
   | +1 :green_heart: |  compile  |   3m 54s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 54s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 47s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  compile  |   3m 55s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  71m  0s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m  4s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  96m 33s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/9/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 8a90d7d97d80 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / 383ffd991a4f67babf218b28c3b9193835c4fa25 |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/9/testReport/ |
   | Max. process+thread count | 1418 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/9/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4484490490

   Run failed because of infra issue, will re-trigger
   ```
   2026-05-18T18:06:26.552Z] ERROR: Caught signal TERM.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4472193186

   @abstractdog , regarding your above question on `test-patch` profile.
   
   Let me share my understanding:
   1. By default, the standard maven-compiler-plugin hides many lint warnings and Yetus relies on doing a diff of the warnings on master vs the warnings on the PR and tells that, PR has introduced 2 more warning or fixed 2 warning for instance. Frankly, I don't see much value of that because in future if JDK25 deprecates a API then this warning diff will flag it. For Tez-1.0.0 community can take a call whether to keep track of warning or remove this profile from codebase. Below table should help in making decision
   2. From my short experience with yetus, it compare things with the baseline master branch against your PR. I couldn't find a way to just run `mvninstall` yetus plugin or `compile` yetus plugin only on PR/patch and ignore master. 
   
   
   Let me clarify a thing:
   1. eliminating YETUS completley, is it acceptable? and move everything to standard Jenkinsfile? We'll lose the yetus summary output and spotbugs/checkstyle/linter which comes out of the box in yetus. Then it will be just like hive project. 
   
   | Feature | Option 1 (Maximum Strictness) | Option 2 (The Sweet Spot) | Option 3 (Maximum Speed) |
   | :--- | :--- | :--- | :--- |
   | **Yetus Phases Enabled** | `mvninstall`, `compile`, `unit` | `compile`, `unit` <br>*(Disabled: `mvninstall`)* | `unit` only <br>*(Disabled: `mvninstall`, `compile`)* |
   | **Step 1: Baseline** | `maven install: master` <br> `master compilation: pre-patch` | `master compilation: pre-patch` | *(Skipped entirely)* |
   | **Step 2: Patch Build** | `maven install: patch` <br> `master compilation: patch` | `master compilation: patch` | *(Skipped entirely)* |
   | **Step 3: Test Execution** | `unit: patch` | `unit: patch` | `unit: patch` |
   | **Estimated Maven Overhead** | ~20 minutes | ~8 minutes | 0 minutes |
   | **Pros** | Maximum safety. Fails early on missing dependencies. | Drops JAR packaging time. Retains `test-patch` profile and early syntax failure detection and tracks compiler warning diffs. | Absolute fastest pipeline. |
   | **Cons** | Very slow. Creates redundant JARs that `unit` doesn't strictly need. | Very slight overhead to count warnings. | **Loses all warning tracking** (e.g., deprecations). Syntax errors fail deep in the `unit` phase instead of failing early.  |
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4474878220

   @abstractdog , I'll create 2 dummy PR to test this jenkins change on TEZ-4708 and TEZ-4711. 
   Both PR will give more confidence in the latest jenkins change and maybe find some early issues if they surface.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4461614212

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 24s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ master Compile Tests _ |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  codespell  |   0m  8s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 34s |  |  The patch does not generate ASF License warnings.  |
   |  |   |   2m 43s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/6/artifact/out/Dockerfile |
   | Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 34092625aa00 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / 383ffd991a4f67babf218b28c3b9193835c4fa25 |
   | Max. process+thread count | 47 (vs. ulimit of 2000) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/6/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255143068


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500

Review Comment:
   why not simply exporting these and getting rid of `SC2034`?



##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then

Review Comment:
   shouldn't it check against strings "unit" and "compile"? like it was done a few lines above
   



##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}
+  done
+}
+
+## @description  File filter to determine which tests to run
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_file_filter
+{
+  local filename=$1
+
+  yetus_debug "Tez file filter: ${filename}"

Review Comment:
   I checked runs #7 and #8 and haven't seen "Tez file filter", am I supposed to see it?



##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}

Review Comment:
   I think the shellcheck warning can be handled by "${module}" "${extra}", couldn't it?



##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{

Review Comment:
   I got used to not having a line break before `{`, looks strange, would you consider doing `function personality_globals {`? if you agree, please take care of the other methods too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255262618


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}

Review Comment:
   yes, will do that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3256695705


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}
+  done
+}
+
+## @description  File filter to determine which tests to run
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_file_filter
+{
+  local filename=$1
+
+  yetus_debug "Tez file filter: ${filename}"

Review Comment:
   Sounds reasonable. Will address it by EOD.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4471938840

   > > * Containerized Yetus Execution: Configured Apache Yetus to run entirely within an isolated Docker container using `build-tools/docker/Dockerfile`  ensuring a reproducible build environment.
   > > * Removed Legacy Scripts: Deleted old custom shell scripts (build-tools/smart-apply-patch.sh and build-tools/test-patch.sh), delegating test orchestration entirely to upstream Apache Yetus.
   > > * Introduced Custom Yetus Personality: Created `dev-support/tez-personality.sh` to centralize and strictly control Tez-specific test execution logic, moving intelligence out of the Jenkinsfile.
   > > * Enforced Global Builds & Unit Tests: Configured the Yetus personality to unconditionally execute the compile and unit test phases on the root module for every PR. This guarantees that cross-module regressions are caught, even for infrastructure-only changes.
   > > * Smart Static Analysis: Optimized `spotbugs, checkstyle, javac, and javadoc` to run "smartly" (only on changed modules) to improve build times, while forcing full-project checks if critical files (like .proto or pom.xml) are modified.
   > > * Improved Artifact Archiving: Added post-build steps to automatically zip and archive all surefire-reports, keeping the Jenkins console clean while making granular test debugging much easier.
   > > 
   > > **Flags REMOVED:**
   > > 
   > > * --skip-dirs=`dev-support`: Removed to allow Yetus to properly analyze and lint changes made to the CI infrastructure scripts themselves.
   > > * --shelldocs=...`/dev-support/bin/shelldocs`: Removed because the path doesn't exists; Use shellcheck bundled in the Docker image.
   > > * --proclimit=5500: Removed from CLI arguments. (Migrated to `PROC_LIMIT` inside `tez-personality.sh`).
   > > * --dockermemlimit=20g: Removed from CLI arguments. (Migrated to `DOCKER_MEMORY` inside `tez-personality.sh`).
   > > * --debug: Removed to reduce log spam in standard CI runs.
   > > 
   > > **Flags ADDED:**
   > > 
   > > * --personality=`.../tez-personality.sh`: Added to load the newly created Tez-specific build and filtering logic.
   > 
   > just for clarity's sake, this is not a new thing, right?
   > 
   > ```
   > Containerized Yetus Execution: Configured Apache Yetus to run entirely within an isolated Docker container using build-tools/docker/Dockerfile ensuring a reproducible build environment.
   > ```
   
   yeah yeah. --docker flag was present earlier as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #498:
URL: https://github.com/apache/tez/pull/498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4490466692

   > Because of https://github.com/apache/yetus/blob/dda07307f7eac00257a33ec61cac86dbde8d1e6d/precommit/src/main/shell/plugins.d/checkstyle.sh#L196-L200 the `test-patch` is hardcoded in checkstyle. Will revert back to `test-patch` from `strict-warnings`.
   > 
   > Sorry for running too many CI 😔
   
   don't be sorry, you are testing CI, that's needed to make these valuable contributions happen! 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4494988127

   @abstractdog , the PR is ready from my side. I think the improvement is good.
   Evidence, for #493 , compare the 2 runs 
   lastet Jenkinsfile: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-500/3/Yetus_20Report/
   Older Jenkinsfile: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-493/9/Yetus_20Report/
   
   | Metric | Older | Newer |
   | :--- | :--- | :--- |
   | **Time Taken** | 242 min | 166 min |
   | **UT Execution Level** | Individual Submodule | Root (Project Level) |
   | **`mvn install` Status** | **Failure** (Older submodule dependency snapshot issues) | **Success (Green)** (Build at project level) |
   
   **All in All we are adding approx 70  min of UT in every PR** . spotbugs, linter, javadoc time will be dependent on PR changes as they are retaining the behaviour of running on submodule level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4471931369

   > * Containerized Yetus Execution: Configured Apache Yetus to run entirely within an isolated Docker container using `build-tools/docker/Dockerfile`  ensuring a reproducible build environment.
   > * Removed Legacy Scripts: Deleted old custom shell scripts (build-tools/smart-apply-patch.sh and build-tools/test-patch.sh), delegating test orchestration entirely to upstream Apache Yetus.
   > * Introduced Custom Yetus Personality: Created `dev-support/tez-personality.sh` to centralize and strictly control Tez-specific test execution logic, moving intelligence out of the Jenkinsfile.
   > * Enforced Global Builds & Unit Tests: Configured the Yetus personality to unconditionally execute the compile and unit test phases on the root module for every PR. This guarantees that cross-module regressions are caught, even for infrastructure-only changes.
   > * Smart Static Analysis: Optimized `spotbugs, checkstyle, javac, and javadoc` to run "smartly" (only on changed modules) to improve build times, while forcing full-project checks if critical files (like .proto or pom.xml) are modified.
   > * Improved Artifact Archiving: Added post-build steps to automatically zip and archive all surefire-reports, keeping the Jenkins console clean while making granular test debugging much easier.
   > 
   > **Flags REMOVED:**
   > 
   > * --skip-dirs=`dev-support`: Removed to allow Yetus to properly analyze and lint changes made to the CI infrastructure scripts themselves.
   > * --shelldocs=...`/dev-support/bin/shelldocs`: Removed because the path doesn't exists; Use shellcheck bundled in the Docker image.
   > * --proclimit=5500: Removed from CLI arguments. (Migrated to `PROC_LIMIT` inside `tez-personality.sh`).
   > * --dockermemlimit=20g: Removed from CLI arguments. (Migrated to `DOCKER_MEMORY` inside `tez-personality.sh`).
   > * --debug: Removed to reduce log spam in standard CI runs.
   > 
   > **Flags ADDED:**
   > 
   > * --personality=`.../tez-personality.sh`: Added to load the newly created Tez-specific build and filtering logic.
   
   
   just for clarity's sake, this is not a new thing, right?
   ```
   Containerized Yetus Execution: Configured Apache Yetus to run entirely within an isolated Docker container using build-tools/docker/Dockerfile ensuring a reproducible build environment.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255267690


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}
+  done
+}
+
+## @description  File filter to determine which tests to run
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_file_filter
+{
+  local filename=$1
+
+  yetus_debug "Tez file filter: ${filename}"

Review Comment:
   --debug flag is removed from Jenkinsfile as it was too much verbose. Once --debug is enabled then this `Tez file filter:` will be printed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4474681267

   > @abstractdog , regarding your above question on `test-patch` profile.
   > 
   > Let me share my understanding:
   > 
   > 1. By default, the standard maven-compiler-plugin hides many lint warnings and Yetus relies on doing a diff of the warnings on master vs the warnings on the PR and tells that, PR has introduced 2 more warning or fixed 2 warning for instance. Frankly, I don't see much value of that because in future if JDK25 deprecates a API then this warning diff will flag it. For Tez-1.0.0 community can take a call whether to keep track of warning or remove this profile from codebase. Below table should help in making decision
   > 2. From my short experience with yetus, it compare things with the baseline master branch against your PR. I couldn't find a way to just run `mvninstall` yetus plugin or `compile` yetus plugin only on PR/patch and ignore master.
   > 
   > Let me clarify a thing:
   > 
   > 1. eliminating YETUS completley, is it acceptable? and move everything to standard Jenkinsfile? We'll lose the yetus summary output and spotbugs/checkstyle/linter which comes out of the box in yetus. Then it will be just like hive project.
   > 
   > Feature	Option 1 (Maximum Strictness)	Option 2 (The Sweet Spot)	Option 3 (Maximum Speed)
   > **Yetus Phases Enabled**	`mvninstall`, `compile`, `unit`	`compile`, `unit`
   > _(Disabled: `mvninstall`)_	`unit` only
   > _(Disabled: `mvninstall`, `compile`)_
   > **Step 1: Baseline**	`maven install: master`
   > `master compilation: pre-patch`	`master compilation: pre-patch`	_(Skipped entirely)_
   > **Step 2: Patch Build**	`maven install: patch`
   > `master compilation: patch`	`master compilation: patch`	_(Skipped entirely)_
   > **Step 3: Test Execution**	`unit: patch`	`unit: patch`	`unit: patch`
   > **Estimated Maven Overhead**	~20 minutes	~8 minutes	0 minutes
   > **Pros**	Maximum safety. Fails early on missing dependencies.	Drops JAR packaging time. Retains `test-patch` profile and early syntax failure detection and tracks compiler warning diffs.	Absolute fastest pipeline.
   > **Cons**	Very slow. Creates redundant JARs that `unit` doesn't strictly need.	Very slight overhead to count warnings.	**Loses all warning tracking** (e.g., deprecations). Syntax errors fail deep in the `unit` phase instead of failing early.
   
   I would go for "Option 1 (Maximum Strictness)", even if I like the optimization in the "Option 2 (The Sweet Spot)":
   
   1. there could be many maven stages between `compile` and `install`, which we should cover (e.g. not sure, but maybe javadoc, etc.): using compile only could end up silently breaking things (and we're going to face them before a release)
   2. by running all the unit tests - as that's what we're trying to achieve now - we vote for full coverage, so we tend to accept longer runtimes for maximum strictness: if it's too bad, we can optimize later, but for now, we can afford this -> given the pace of Tez project, a few longer runs are absolutely okay, so from this point of view, I personally don't care about "Creates redundant JARs that unit doesn't strictly need."
   
   
   regarding discontinuing Yetus: this points too far and needs community consensus: I mean that's okay, but the PR hits the local optima, I'm really satisfied with short-/mid-term
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4461411649

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 46s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ master Compile Tests _ |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  codespell  |   0m  8s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 34s |  |  The patch does not generate ASF License warnings.  |
   |  |   |   3m  3s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/5/artifact/out/Dockerfile |
   | Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 5c25ab76080c 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / 383ffd991a4f67babf218b28c3b9193835c4fa25 |
   | Max. process+thread count | 47 (vs. ulimit of 2000) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/5/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4495200796

   Thanks, once https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/ completes we can get this merged 👍🏻


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4485970004

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  2s |  |  master passed  |
   | +1 :green_heart: |  compile  |   1m 54s |  |  master passed  |
   | +1 :green_heart: |  javadoc  |   1m 27s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 56s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 45s |  |  The patch generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3)  |
   | +1 :green_heart: |  compile  |   3m 55s |  |  the patch passed  |
   | -1 :x: |  javac  |   3m 55s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/12/artifact/out/results-compile-javac-root.txt) |  root generated 742 new + 0 unchanged - 34 fixed = 742 total (was 34)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   | +1 :green_heart: |  javadoc  |   1m 23s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  71m  6s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 57s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  93m 29s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/12/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs javac javadoc xmllint |
   | uname | Linux fe0c311cd56e 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / d0fa41151f2ccdd632210689fa6bc1a184f9c0a2 |
   | Default Java | Eclipse Adoptium-21.0.11+10-LTS |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/12/testReport/ |
   | Max. process+thread count | 1359 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/12/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4486036240

   Because of https://github.com/apache/yetus/blob/dda07307f7eac00257a33ec61cac86dbde8d1e6d/precommit/src/main/shell/plugins.d/checkstyle.sh#L196-L200 the `test-patch` is hardcoded in checkstyle. Will revert back to `test-patch` from `strict-warnings`.
   
   Sorry for running too many CI 😔


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4472059687

   > So, between Run7 and Run8 the the build time has reduced from:
   > 
   > * Run 7 Total Time: 131m 18s
   > * Run 8 Total Time: 94m 16s
   >   27 minutes were saved from the CI time because of dropping the `-Ptest-patch` during the unit test phase (which dropped the unit test duration from 95m to 68m)
   > 
   > **Point to Ponder:** Yetus is running build 4 times:
   > 
   > 1. maven install: master: `mvn clean install` on master branch.
   > 2. master compilation: pre-patch:  `mvn clean test-compile -Ptest-patch` on master branch, Uses `test-patch` profile to count the warnings.
   > 3. maven install: patch: `mvn clean install` runs on PR/patch (Master + PR changes)
   > 4. master compilation: patch: ` mvn clean test-compile -Ptest-patch` runs on PR/patch (Master + PR changes) to check for the warnings.
   > 
   > Here is breakdown of each `mvn clean command`:
   > 
   > 1. maven install: master (Build 1) : Elapsed: 6m 55s
   > 2. master compilation: pre-patch (Build 2): Elapsed: 3m 55s
   > 3. maven install: patch (Build 3): Elapsed: 5m 55s
   > 4. master compilation: patch (Build 4): Elapsed: 3m 54s
   > 
   > Question, Ques1 -> do we need compile plugin which is being used for warning diff calculation? but it is taking approx 8 min i.e. 2 and 4. Ques2 -> We can remove Step 1 and 3 as well by doing `YETUS_ARGS+=("--plugins=all,-mvninstall")` , this is just putting the jars in .m2
   > 
   > **I think keeping the compile stage makes more sense as it takes less time and check the project for sytax error. Also, `unit: patch` will anyway build and test the whole project.**
   
   @Aggarwal-Raghav : left some syntax comments, also thought about your questions here:
   
   what about shooting for all with an "install" command, so having 2 stages (master + patch):
   ```
   mvn clean install -Ptest-patch
   ```
   
   and taking one more look, what value does the `test-patch` profile bring at all?
   https://github.com/apache/tez/blob/383ffd991a4f67babf218b28c3b9193835c4fa25/pom.xml#L1236-L1256
   is it just for having custom compiler arguments, other than what we have in case of a simple `mvn clean install` command?
   
   also, I guess the `test-patch` naming was brought by TEZ-1313 in the early years of the project (the name must have matched the yetus terminology): I'm just thinking aloud, considering that `test-patch.sh` is removed from the codebase now, we can also rename/refactor profiles if they make more sense (not necessary, just an idea)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4490599513

   1. Retained the `test-patch` profile as it was failing because in https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/12/artifact/out/branch-compile-root.txt `The requested profile "strict-warnings" could not be activated because it does not exist.` as this change is not merged in master. Once this PR is merged then assuming it will work for following PR but to not take that chance I have retained `test-patch`. Let me know your thoughts on this.
   
   2. Retained mvninstall as given reasoning in above comment and ensured it also runs fully on the project.
   3. Added tail for UT to see the progress and find out from console logs where the test hangs if any, please check #500 
   4. Added timeout for yetus. That it should complete in 3 hr and jenkins pipeline hard stop/ abort at 4 hr. I think in very very rare case 4hrs will be triggered mainly when post action example zip is taking time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4495216558

   > > Thanks, once https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/ completes we can get this merged 👍🏻
   > 
   > oops, already merged...let's follow that run in case of any problems :)
   
   yes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4495749978

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 29s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  3s |  |  master passed  |
   | +1 :green_heart: |  compile  |   3m 53s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  0s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 48s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  compile  |   3m 53s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  67m 58s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 57s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  89m 29s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 971ab2b9b1c0 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / d0fa41151f2ccdd632210689fa6bc1a184f9c0a2 |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/testReport/ |
   | Max. process+thread count | 1319 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/14/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255273994


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then

Review Comment:
   both == unit and == "unit" work the same way when there are no spaces or special characters in the word. But yes, putting quotes will help in unwanted globbing. Will update.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4467949389

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 37s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m 55s |  |  master passed  |
   | +1 :green_heart: |  compile  |   3m 55s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 55s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 47s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  compile  |   3m 54s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  68m 47s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 57s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  94m 16s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/8/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux c611ee8d3db9 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / 383ffd991a4f67babf218b28c3b9193835c4fa25 |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/8/testReport/ |
   | Max. process+thread count | 1304 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/8/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4468039385

   So, between Run7 and Run8 the  the build time has reduced from: 
      * Run 7 Total Time: 131m 18s
      * Run 8 Total Time: 94m 16s
   27 minutes were saved from the CI time because of dropping the `-Ptest-patch` during the unit test phase (which dropped the unit test duration from 95m to 68m)
      
   **Point to Ponder:**
   Yetus is running build 4 times: 
   1.  maven install: master: `mvn clean install` on master branch.
   2. master compilation: pre-patch:  `mvn clean test-compile -Ptest-patch` on master branch, Uses `test-patch` profile to count the warnings.
   3. maven install: patch: `mvn clean install` runs on PR/patch (Master + PR changes) 
   4. master compilation: patch: ` mvn clean test-compile -Ptest-patch` runs on PR/patch (Master + PR changes) to check for the warnings.
   
   Here is breakdown of each `mvn clean command`:
   1. maven install: master (Build 1) : Elapsed: 6m 55s
   2. master compilation: pre-patch (Build 2): Elapsed: 3m 55s
   3. maven install: patch (Build 3): Elapsed: 5m 55s
   4. master compilation: patch (Build 4): Elapsed: 3m 54s
   
   Question, 
   Ques1 -> do we need compile plugin which is being used for warning diff calculation? but it is taking approx 8 min  i.e. 2 and 4. 
   Ques2 -> We can remove Step 1 and 3 as well by doing `YETUS_ARGS+=("--plugins=all,-mvninstall")` , this is just putting the jars in .m2 
   
   **I think keeping the compile stage makes more sense as it takes less time and check the project for sytax error. Also, `unit: patch` will anyway build and test the whole project.**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4484927871

   In the latest run i.e 11, it is doing `maven install: master` and `maven install: patch` despite `YETUS_ARGS+=("--plugins=all,-mvninstall")`, after checking yetus, 
   for spotbugs: https://github.com/apache/yetus/blob/rel/0.15.1/precommit/src/main/shell/plugins.d/spotbugs.sh#L55-L57 and 
   for checkstyle: https://github.com/apache/yetus/blob/rel/0.15.1/precommit/src/main/shell/plugins.d/checkstyle.sh#L90-L95 
   it is adding mvninstall forcefully.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4485058181

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 59s |  |  master passed  |
   | +1 :green_heart: |  compile  |   1m 54s |  |  master passed  |
   | +1 :green_heart: |  javadoc  |   1m 28s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 52s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 48s |  |  The patch generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3)  |
   | +1 :green_heart: |  compile  |   3m 54s |  |  the patch passed  |
   | -1 :x: |  javac  |   3m 54s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/11/artifact/out/results-compile-javac-root.txt) |  root generated 742 new + 0 unchanged - 34 fixed = 742 total (was 34)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  69m 59s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  94m 17s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/11/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs javac javadoc xmllint |
   | uname | Linux 01193934385a 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / d0fa41151f2ccdd632210689fa6bc1a184f9c0a2 |
   | Default Java | Eclipse Adoptium-21.0.11+10-LTS |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/11/testReport/ |
   | Max. process+thread count | 1468 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/11/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3256665002


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500
+
+  # Force-enable core verification plugins
+  add_test compile
+  add_test unit
+}
+
+## @description  Module selection
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_modules
+{
+  local repostatus=$1
+  local testtype=$2
+  local extra=""
+  local MODULES=("${CHANGED_MODULES[@]}")
+
+  yetus_debug "Personality: ${repostatus} ${testtype}"
+
+  clear_personality_queue
+
+  extra="-Ptools"
+
+  # Apply strict linting profile for all phases except unit tests
+  if [[ ${testtype} != "unit" ]]; then
+    extra="${extra} -Ptest-patch"
+  fi
+
+  # Execute core compilation and unit tests globally (project root)
+  if [[ ${testtype} == unit || ${testtype} == compile ]]; then
+    yetus_debug "Forcing root module for ${testtype}"
+    MODULES=(.)
+  fi
+
+  if [[ ${testtype} == spotbugs ]]; then
+    extra="${extra} -Pspotbugs"
+  fi
+
+  for module in "${MODULES[@]}"; do
+    # shellcheck disable=SC2086
+    personality_enqueue_module ${module} ${extra}
+  done
+}
+
+## @description  File filter to determine which tests to run
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_file_filter
+{
+  local filename=$1
+
+  yetus_debug "Tez file filter: ${filename}"

Review Comment:
   thanks, I'm fine with that, so in case of an evil pre-commit issue, `--debug` can be added to the jenkins pipeline script while manually Replaying the job
   
   <img width="721" height="572" alt="Screenshot 2026-05-18 at 11 30 56" src="https://github.com/user-attachments/assets/6da9c17f-c48c-45ff-8a2e-0fe7a803718e" />
   
   I'm usually against comments containing code, but in this case, a commented-out `--debug` option could be really helpful, what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255269138


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{
+  # shellcheck disable=SC2034
+  PROJECT_NAME=tez
+  # shellcheck disable=SC2034
+  BUILDTOOL=maven
+  # shellcheck disable=SC2034
+  PATCH_BRANCH_DEFAULT=master
+  # shellcheck disable=SC2034
+  JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+  # shellcheck disable=SC2034
+  GITHUB_REPO="apache/tez"
+
+  # Increase memory for maven
+  # shellcheck disable=SC2034
+  MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx4g -XX:+UseG1GC"}"
+
+  # Default Yetus settings
+  # shellcheck disable=SC2034
+  DOCKER_MEMORY="20g"
+  # shellcheck disable=SC2034
+  PROC_LIMIT=5500

Review Comment:
   Yes, will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4461962207

   Phew! Now, it is running build and UT. The changes are inspired from apache phoenix project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #498:
URL: https://github.com/apache/tez/pull/498#discussion_r3255268148


##########
dev-support/tez-personality.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Yetus personality for Apache Tez
+
+personality_plugins "all"
+
+## @description  Globals for this personality
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function personality_globals
+{

Review Comment:
   ACK. will handle it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4474913602

   > > > > @abstractdog , regarding your above question on `test-patch` profile.
   > > > > Let me share my understanding:
   > > > > 
   > > > > 1. By default, the standard maven-compiler-plugin hides many lint warnings and Yetus relies on doing a diff of the warnings on master vs the warnings on the PR and tells that, PR has introduced 2 more warning or fixed 2 warning for instance. Frankly, I don't see much value of that because in future if JDK25 deprecates a API then this warning diff will flag it. For Tez-1.0.0 community can take a call whether to keep track of warning or remove this profile from codebase. Below table should help in making decision
   > > > > 2. From my short experience with yetus, it compare things with the baseline master branch against your PR. I couldn't find a way to just run `mvninstall` yetus plugin or `compile` yetus plugin only on PR/patch and ignore master.
   > > > > 
   > > > > Let me clarify a thing:
   > > > > 
   > > > > 1. eliminating YETUS completley, is it acceptable? and move everything to standard Jenkinsfile? We'll lose the yetus summary output and spotbugs/checkstyle/linter which comes out of the box in yetus. Then it will be just like hive project.
   > > > > 
   > > > > Feature	Option 1 (Maximum Strictness)	Option 2 (The Sweet Spot)	Option 3 (Maximum Speed)
   > > > > **Yetus Phases Enabled**	`mvninstall`, `compile`, `unit`	`compile`, `unit`
   > > > > _(Disabled: `mvninstall`)_	`unit` only
   > > > > _(Disabled: `mvninstall`, `compile`)_
   > > > > **Step 1: Baseline**	`maven install: master`
   > > > > `master compilation: pre-patch`	`master compilation: pre-patch`	_(Skipped entirely)_
   > > > > **Step 2: Patch Build**	`maven install: patch`
   > > > > `master compilation: patch`	`master compilation: patch`	_(Skipped entirely)_
   > > > > **Step 3: Test Execution**	`unit: patch`	`unit: patch`	`unit: patch`
   > > > > **Estimated Maven Overhead**	~20 minutes	~8 minutes	0 minutes
   > > > > **Pros**	Maximum safety. Fails early on missing dependencies.	Drops JAR packaging time. Retains `test-patch` profile and early syntax failure detection and tracks compiler warning diffs.	Absolute fastest pipeline.
   > > > > **Cons**	Very slow. Creates redundant JARs that `unit` doesn't strictly need.	Very slight overhead to count warnings.	**Loses all warning tracking** (e.g., deprecations). Syntax errors fail deep in the `unit` phase instead of failing early.
   > > > 
   > > > 
   > > > I would go for "Option 1 (Maximum Strictness)", even if I like the optimization in the "Option 2 (The Sweet Spot)":
   > > > 
   > > > 1. there could be many maven stages between `compile` and `install`, which we should cover (e.g. not sure, but maybe javadoc, etc.): using compile only could end up silently breaking things (and we're going to face them before a release)
   > > > 2. by running all the unit tests - as that's what we're trying to achieve now - we vote for full coverage, so we tend to accept longer runtimes for maximum strictness: if it's too bad, we can optimize later, but for now, we can afford this -> given the pace of Tez project, a few longer runs are absolutely okay, so from this point of view, I personally don't care about "Creates redundant JARs that unit doesn't strictly need."
   > > > 
   > > > regarding discontinuing Yetus: this points too far and needs community consensus: I mean that's okay, but the PR hits the local optima, I'm really satisfied with short-/mid-term
   > > 
   > > 
   > > hm, wait, I just realized again that github actions cover the `mvn clean install` scenario by running:
   > > https://github.com/apache/tez/blob/d0fa41151f2ccdd632210689fa6bc1a184f9c0a2/.github/workflows/build.yml#L49-L50
   > > 
   > > ```
   > > mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > which is even better, as it runs on multiple platforms
   > > considering the minimal time difference between these options (on my laptop):
   > > ```
   > > mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   > > ...
   > > [INFO] Total time:  50.257 s
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > ```
   > > mvn --batch-mode --no-transfer-progress clean install -DskipTests
   > > ...
   > > [INFO] Total time:  49.779 s
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > so what about running full mvn install in github actions, and going for "Option 2 (The Sweet Spot)" in yetus?
   > 
   > To clarify, you'll need to configure GitHub Actions to run automatically on pull requests. I assume you're currently initiating them manually for each pull request..
   
   yeah, and I think we should run the build workflow automatically, so it still points in the same direction


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4474897976

   > > > @abstractdog , regarding your above question on `test-patch` profile.
   > > > Let me share my understanding:
   > > > 
   > > > 1. By default, the standard maven-compiler-plugin hides many lint warnings and Yetus relies on doing a diff of the warnings on master vs the warnings on the PR and tells that, PR has introduced 2 more warning or fixed 2 warning for instance. Frankly, I don't see much value of that because in future if JDK25 deprecates a API then this warning diff will flag it. For Tez-1.0.0 community can take a call whether to keep track of warning or remove this profile from codebase. Below table should help in making decision
   > > > 2. From my short experience with yetus, it compare things with the baseline master branch against your PR. I couldn't find a way to just run `mvninstall` yetus plugin or `compile` yetus plugin only on PR/patch and ignore master.
   > > > 
   > > > Let me clarify a thing:
   > > > 
   > > > 1. eliminating YETUS completley, is it acceptable? and move everything to standard Jenkinsfile? We'll lose the yetus summary output and spotbugs/checkstyle/linter which comes out of the box in yetus. Then it will be just like hive project.
   > > > 
   > > > Feature	Option 1 (Maximum Strictness)	Option 2 (The Sweet Spot)	Option 3 (Maximum Speed)
   > > > **Yetus Phases Enabled**	`mvninstall`, `compile`, `unit`	`compile`, `unit`
   > > > _(Disabled: `mvninstall`)_	`unit` only
   > > > _(Disabled: `mvninstall`, `compile`)_
   > > > **Step 1: Baseline**	`maven install: master`
   > > > `master compilation: pre-patch`	`master compilation: pre-patch`	_(Skipped entirely)_
   > > > **Step 2: Patch Build**	`maven install: patch`
   > > > `master compilation: patch`	`master compilation: patch`	_(Skipped entirely)_
   > > > **Step 3: Test Execution**	`unit: patch`	`unit: patch`	`unit: patch`
   > > > **Estimated Maven Overhead**	~20 minutes	~8 minutes	0 minutes
   > > > **Pros**	Maximum safety. Fails early on missing dependencies.	Drops JAR packaging time. Retains `test-patch` profile and early syntax failure detection and tracks compiler warning diffs.	Absolute fastest pipeline.
   > > > **Cons**	Very slow. Creates redundant JARs that `unit` doesn't strictly need.	Very slight overhead to count warnings.	**Loses all warning tracking** (e.g., deprecations). Syntax errors fail deep in the `unit` phase instead of failing early.
   > > 
   > > I would go for "Option 1 (Maximum Strictness)", even if I like the optimization in the "Option 2 (The Sweet Spot)":
   > > 
   > > 1. there could be many maven stages between `compile` and `install`, which we should cover (e.g. not sure, but maybe javadoc, etc.): using compile only could end up silently breaking things (and we're going to face them before a release)
   > > 2. by running all the unit tests - as that's what we're trying to achieve now - we vote for full coverage, so we tend to accept longer runtimes for maximum strictness: if it's too bad, we can optimize later, but for now, we can afford this -> given the pace of Tez project, a few longer runs are absolutely okay, so from this point of view, I personally don't care about "Creates redundant JARs that unit doesn't strictly need."
   > > 
   > > regarding discontinuing Yetus: this points too far and needs community consensus: I mean that's okay, but the PR hits the local optima, I'm really satisfied with short-/mid-term
   > 
   > hm, wait, I just realized again that github actions cover the `mvn clean install` scenario by running:
   > https://github.com/apache/tez/blob/d0fa41151f2ccdd632210689fa6bc1a184f9c0a2/.github/workflows/build.yml#L49-L50
   > ```
   > mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   > ```
   > which is even better, as it runs on multiple platforms
   > 
   > considering the minimal time difference between these options (on my laptop):
   > ```
   > mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   > ...
   > [INFO] Total time:  50.257 s
   > ```
   > 
   > ```
   > mvn --batch-mode --no-transfer-progress clean install -DskipTests
   > ...
   > [INFO] Total time:  49.779 s
   > ```
   > 
   > so what about running full mvn install in github actions, and going for "Option 2 (The Sweet Spot)" in yetus?
   
   But then you need to enable GitHub actions by default on the PR. I'm guessing your are triggering it manually on the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4474879004

   > > @abstractdog , regarding your above question on `test-patch` profile.
   > > Let me share my understanding:
   > > 
   > > 1. By default, the standard maven-compiler-plugin hides many lint warnings and Yetus relies on doing a diff of the warnings on master vs the warnings on the PR and tells that, PR has introduced 2 more warning or fixed 2 warning for instance. Frankly, I don't see much value of that because in future if JDK25 deprecates a API then this warning diff will flag it. For Tez-1.0.0 community can take a call whether to keep track of warning or remove this profile from codebase. Below table should help in making decision
   > > 2. From my short experience with yetus, it compare things with the baseline master branch against your PR. I couldn't find a way to just run `mvninstall` yetus plugin or `compile` yetus plugin only on PR/patch and ignore master.
   > > 
   > > Let me clarify a thing:
   > > 
   > > 1. eliminating YETUS completley, is it acceptable? and move everything to standard Jenkinsfile? We'll lose the yetus summary output and spotbugs/checkstyle/linter which comes out of the box in yetus. Then it will be just like hive project.
   > > 
   > > Feature	Option 1 (Maximum Strictness)	Option 2 (The Sweet Spot)	Option 3 (Maximum Speed)
   > > **Yetus Phases Enabled**	`mvninstall`, `compile`, `unit`	`compile`, `unit`
   > > _(Disabled: `mvninstall`)_	`unit` only
   > > _(Disabled: `mvninstall`, `compile`)_
   > > **Step 1: Baseline**	`maven install: master`
   > > `master compilation: pre-patch`	`master compilation: pre-patch`	_(Skipped entirely)_
   > > **Step 2: Patch Build**	`maven install: patch`
   > > `master compilation: patch`	`master compilation: patch`	_(Skipped entirely)_
   > > **Step 3: Test Execution**	`unit: patch`	`unit: patch`	`unit: patch`
   > > **Estimated Maven Overhead**	~20 minutes	~8 minutes	0 minutes
   > > **Pros**	Maximum safety. Fails early on missing dependencies.	Drops JAR packaging time. Retains `test-patch` profile and early syntax failure detection and tracks compiler warning diffs.	Absolute fastest pipeline.
   > > **Cons**	Very slow. Creates redundant JARs that `unit` doesn't strictly need.	Very slight overhead to count warnings.	**Loses all warning tracking** (e.g., deprecations). Syntax errors fail deep in the `unit` phase instead of failing early.
   > 
   > I would go for "Option 1 (Maximum Strictness)", even if I like the optimization in the "Option 2 (The Sweet Spot)":
   > 
   > 1. there could be many maven stages between `compile` and `install`, which we should cover (e.g. not sure, but maybe javadoc, etc.): using compile only could end up silently breaking things (and we're going to face them before a release)
   > 2. by running all the unit tests - as that's what we're trying to achieve now - we vote for full coverage, so we tend to accept longer runtimes for maximum strictness: if it's too bad, we can optimize later, but for now, we can afford this -> given the pace of Tez project, a few longer runs are absolutely okay, so from this point of view, I personally don't care about "Creates redundant JARs that unit doesn't strictly need."
   > 
   > regarding discontinuing Yetus: this points too far and needs community consensus: I mean that's okay, but the PR hits the local optima, I'm really satisfied with short-/mid-term
   
   hm, wait, I just realized again that github actions cover the `mvn clean install` scenario by running:
   https://github.com/apache/tez/blob/d0fa41151f2ccdd632210689fa6bc1a184f9c0a2/.github/workflows/build.yml#L49-L50
   ```
   mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   ```
   which is even better, as it runs on multiple platforms
   
   considering the minimal time difference between these options (on my laptop):
   ```
   mvn --batch-mode --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true
   ...
   [INFO] Total time:  50.257 s
   ```
   
   ```
   mvn --batch-mode --no-transfer-progress clean install -DskipTests
   ...
   [INFO] Total time:  49.779 s
   ```
   
   so what about running full mvn install in github actions, and going for "Option 2 (The Sweet Spot)" in yetus?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4490395349

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  3s |  |  master passed  |
   | +1 :green_heart: |  compile  |   3m 54s |  |  master passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 56s |  |  the patch passed  |
   | +1 :green_heart: |  codespell  |   1m 46s |  |  The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)  |
   | +1 :green_heart: |  compile  |   3m 55s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  |  The patch generated 0 new + 0 unchanged - 150 fixed = 0 total (was 150)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  71m 37s |  |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 56s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  93m  9s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/13/artifact/out/Dockerfile |
   | Optional Tests | dupname compile unit asflicense codespell detsecrets shellcheck shelldocs |
   | uname | Linux 245361ce0a98 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | tez-personality.sh |
   | git revision | master / d0fa41151f2ccdd632210689fa6bc1a184f9c0a2 |
   |  Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/13/testReport/ |
   | Max. process+thread count | 2105 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-498/13/console |
   | versions | git=2.43.0 maven=3.9.15 codespell=2.4.1 shellcheck=0.11.0 |
   | Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] TEZ-4718: Modernize Jenkins and Yetus integration to ensure full build and UT run on every PR [tez]

Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #498:
URL: https://github.com/apache/tez/pull/498#issuecomment-4467618246

   @abstractdog , the UT ran without the *.java extension changes. I would appreciate a feedback on this behaviour.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org