You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "okumin (via GitHub)" <gi...@apache.org> on 2025/02/28 09:09:24 UTC
[PR] [WIP] TEZ-4604: tez-mapreduce does not delete files under staging directory [tez]
okumin opened a new pull request, #395:
URL: https://github.com/apache/tez/pull/395
https://issues.apache.org/jira/browse/TEZ-4604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] TEZ-4604: tez-mapreduce does not delete files under staging directory [tez]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #395:
URL: https://github.com/apache/tez/pull/395#discussion_r1975090250
##########
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
##########
@@ -623,6 +623,7 @@ public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials ts)
try {
dagAMConf.set(TezConfiguration.TEZ_AM_STAGING_DIR,
jobSubmitDir);
+ dagAMConf.setBoolean(TezConfiguration.TEZ_AM_STAGING_BASE_DIR_CLEANUP, true);
Review Comment:
YARNRunner is a glue code between a MapReduce Job and Tez, implementing `ClientProtocol`. So, the client code of YARNRunner is Apache Hadoop.
`ClientProtocol` doesn't have an API to declare that a specific job has been completed. If we resolve this issue on the client side, we have to add new APIs to Apache Hadoop. That's why I added a new param and handled the issue on Apache Tez side.
I'm not confident that this approach is the best. I'd appreciate it if someone could give me a better idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4604: tez-mapreduce does not delete files under staging directory [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #395:
URL: https://github.com/apache/tez/pull/395#issuecomment-2690379162
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 7s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 10s | master passed |
| +1 :green_heart: | compile | 2m 23s | master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | compile | 2m 12s | master passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | checkstyle | 2m 3s | master passed |
| +1 :green_heart: | javadoc | 1m 58s | master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javadoc | 1m 47s | master passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +0 :ok: | spotbugs | 0m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 10s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 29s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javac | 1m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 23s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | javac | 1m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | findbugs | 3m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 13s | tez-api in the patch passed. |
| +1 :green_heart: | unit | 1m 22s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 5m 6s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 45m 32s | tez-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 123m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/395 |
| JIRA Issue | TEZ-4604 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 16c4867b4566 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 82b3476e0 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/1/testReport/ |
| Max. process+thread count | 1174 (vs. ulimit of 5500) |
| modules | C: tez-api tez-mapreduce tez-dag tez-tests U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/1/console |
| versions | git=2.34.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4604: tez-mapreduce does not delete files under staging directory [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #395:
URL: https://github.com/apache/tez/pull/395#issuecomment-2743531914
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 22m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 3s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 58s | master passed |
| +1 :green_heart: | compile | 2m 27s | master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | compile | 2m 19s | master passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | checkstyle | 2m 5s | master passed |
| +1 :green_heart: | javadoc | 2m 1s | master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javadoc | 1m 51s | master passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +0 :ok: | spotbugs | 0m 46s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 13s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 30s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javac | 1m 30s | the patch passed |
| +1 :green_heart: | compile | 1m 22s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | javac | 1m 22s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| +1 :green_heart: | findbugs | 3m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 11s | tez-api in the patch passed. |
| +1 :green_heart: | unit | 1m 22s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 5m 5s | tez-dag in the patch passed. |
| -1 :x: | unit | 51m 37s | tez-tests in the patch failed. |
| +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. |
| | | 123m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/395 |
| JIRA Issue | TEZ-4604 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 3f45da27f7c5 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 3aa2aa18e |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~22.04-b06 |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/artifact/out/patch-unit-tez-tests.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/testReport/ |
| Max. process+thread count | 1303 (vs. ulimit of 5500) |
| modules | C: tez-api tez-mapreduce tez-dag tez-tests U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/console |
| versions | git=2.34.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org