You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org> on 2026/01/27 18:26:51 UTC
[PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath [tez]
Aggarwal-Raghav opened a new pull request, #458:
URL: https://github.com/apache/tez/pull/458
Please check screenshots in [TEZ-4685](https://issues.apache.org/jira/browse/TEZ-4685)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #458:
URL: https://github.com/apache/tez/pull/458#issuecomment-3922687165
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 14m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 11m 10s | | master passed |
| +1 :green_heart: | compile | 0m 45s | | master passed |
| +1 :green_heart: | checkstyle | 1m 5s | | master passed |
| +1 :green_heart: | javadoc | 0m 35s | | master passed |
| +0 :ok: | spotbugs | 2m 28s | | tez-dag in master has 749 extant spotbugs warnings. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | codespell | 0m 57s | | No new issues. |
| +1 :green_heart: | compile | 0m 32s | | the patch passed |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 49s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 33s | | tez-dag in the patch passed. |
| +1 :green_heart: | asflicense | 0m 12s | | The patch does not generate ASF License warnings. |
| | | 41m 36s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/458 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile |
| uname | Linux 2fa494d80081 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-agent/workspace/tez-multibranch_PR-458/src/.yetus/personality.sh |
| git revision | master / 80f79e85a4af1d07e98a9b2a580b435b26f94bff |
| Default Java | Ubuntu-21.0.10+7-Ubuntu-124.04 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/testReport/ |
| Max. process+thread count | 238 (vs. ulimit of 5500) |
| modules | C: tez-dag U: tez-dag |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/console |
| versions | git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.4.1 |
| Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2832984544
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
hm, that's a good question: I guess here is where we might want to get back to the original motivation: whether a standalone TezAM docker container needs a tez-site.xml on its classpath (and if so, could it read) or if there is another way to configure it... isn't [TEZ-4014](https://issues.apache.org/jira/browse/TEZ-4014) or some earlier work already taking care of this? what do you think @Aggarwal-Raghav ?
https://github.com/apache/tez/blob/f09ba7faffff18e3197a4963764a34b69a22d656/tez-common/src/main/java/org/apache/tez/common/TezUtilsInternal.java#L100-L105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2832643997
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
This should be kept somehow in FrameworkUtils, like `FrameworkUtils.getInitialConfiguration()`
because we might want to hide the details of this "hack" from DAGAppMaster.
Why I call it a hack: it's not because the solution is bad, it's because of the current design, which leads to a catch-22:
1. we need a Configuration
2. for that we need to know the framework mode
3. to know the framework mode and get framework client/server classes we need a Configuration object: `getFrameworkService(conf)`
so we cannot do anything better than to achieve original Configuration by the value stored in environment...which is fine, but DAGAppMaster's already overwhelmed logic should not contain such a thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #458:
URL: https://github.com/apache/tez/pull/458#issuecomment-3806892990
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 56s | | master passed |
| +1 :green_heart: | compile | 0m 24s | | master passed |
| +1 :green_heart: | checkstyle | 0m 51s | | master passed |
| +1 :green_heart: | javadoc | 0m 24s | | master passed |
| -1 :x: | spotbugs | 1m 3s | [/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/artifact/out/branch-spotbugs-tez-dag.txt) | tez-dag in master failed. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | codespell | 0m 27s | | No new issues. |
| +1 :green_heart: | compile | 0m 18s | | the patch passed |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 10s | | the patch passed |
| -1 :x: | spotbugs | 0m 17s | [/patch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/artifact/out/patch-spotbugs-tez-dag.txt) | tez-dag in the patch failed. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 25s | | tez-dag in the patch passed. |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 20m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/458 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile |
| uname | Linux cdb2f8f4b9fb 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-458/src/.yetus/personality.sh |
| git revision | master / f2eba560ca600303e51d14aed6eaffbc6022ccf0 |
| Default Java | Ubuntu-21.0.9+10-Ubuntu-124.04 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/testReport/ |
| Max. process+thread count | 246 (vs. ulimit of 5500) |
| modules | C: tez-dag U: tez-dag |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/console |
| versions | git=2.43.0 maven=3.8.7 codespell=2.4.1 |
| Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #458:
URL: https://github.com/apache/tez/pull/458#issuecomment-3922476624
@ayushtkn / @abstractdog , I have updated the patch. Can you provide your feedback on this implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2832984544
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
hm, that's a good question: I guess here is where we might want to get back to the original motivation: whether a standalone TezAM docker container needs a tez-site.xml on its classpath (and if so, could it read) or if there is another way to configure it... isn't [TEZ-4014](https://issues.apache.org/jira/browse/TEZ-4014) already taking care of this? what do you think @Aggarwal-Raghav ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2832643997
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
This should be kept somehow in FrameworkUtils, like `FrameworkUtils.getInitialConfiguration()`
because we might want to hide the details of this "hack" from DAGAppMaster.
Why I call it a hack: it's not because the solution is bad, it's because of the current design, which is:
1. we need a Configuration
2. for that we need to know the framework mode
3. the framework mode and classes needs a Configuration object already `getFrameworkService(conf)`
so we cannot do anything better than to achieve original Configuration by the value stored in environment...which is fine, but DAGAppMaster's already overwhelmed logic should not contain such a thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav closed pull request #458: TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode
URL: https://github.com/apache/tez/pull/458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2836336896
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
Thanks, @abstractdog, for pointing out [TEZ-4014](https://issues.apache.org/jira/browse/TEZ-4014).It works well, provided we pass `TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` as an environment variable (using the `--env-file` or `-e` flag in the `docker run` command) and ensure `tez-site.xml` is in the classpath.
Without this environment variable, it defaults back to YARN mode. I might have missed passing the `-e` flag initially while working on `TEZ-4682` stared using with `new TezConfiguration()` as in current TEZ-4682 PR https://github.com/apache/tez/commit/551075bc420baab15ce3f3df1f4445b4c334cab8 and never checked back :-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2832811546
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
@abstractdog do you have pointer what is different in `STANDALONE_ZOOKEEPER` that it doesn't work here but does work in `YARN` mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2833389614
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
I'll get back on this TEZ-4014 analysis in couple of days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode [tez]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #458:
URL: https://github.com/apache/tez/pull/458#discussion_r2836337358
##########
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##########
@@ -2429,7 +2430,10 @@ public static void main(String[] args) {
Objects.requireNonNull(appSubmitTimeStr,
ApplicationConstants.APP_SUBMIT_TIME_ENV + " is null");
- Configuration conf = new Configuration();
+ Configuration conf =
+ STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+ ? new TezConfiguration()
+ : new Configuration();
Review Comment:
I'll close this JIRA then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org