You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "maheshrajus (via GitHub)" <gi...@apache.org> on 2025/11/24 07:51:25 UTC
[PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
maheshrajus opened a new pull request, #445:
URL: https://github.com/apache/tez/pull/445
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3580685042
I still feel the test case is shady in its current form, and I'm afraid this fix won't make it any clearer
having this magic:
```
long requestSize = (long) (0.4f * inMemThreshold) + 100L;
```
requestSize should be greater than inMemThreshold, which is a lower bound, also I guess it has an upper bound constraint (jvm memory, or whatever), am I right to assume that?
could this line be clearer to comply with those constraints and work in every scenario (regardless of the jvmMax)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "maheshrajus (via GitHub)" <gi...@apache.org>.
maheshrajus commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3615392212
@abstractdog I will make requestSize simple instead of jvm memory and "(long) (0.4f * inMemThreshold) + 100L" checks. Lower bound value already tested in existed test cases. i will put requestSize size more than inMemThreshold and test the scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "maheshrajus (via GitHub)" <gi...@apache.org>.
maheshrajus commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3625326900
@abstractdog Can you review and approve the changes ? thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3569562107
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 11m 56s | | master passed |
| +1 :green_heart: | compile | 0m 37s | | master passed |
| +1 :green_heart: | checkstyle | 1m 4s | | master passed |
| +1 :green_heart: | javadoc | 0m 45s | | master passed |
| +0 :ok: | spotbugs | 2m 0s | | tez-runtime-library in master has 241 extant spotbugs warnings. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | codespell | 0m 28s | | No new issues. |
| +1 :green_heart: | compile | 0m 23s | | the patch passed |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 13s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 6m 0s | | tez-runtime-library in the patch passed. |
| +1 :green_heart: | asflicense | 0m 13s | | The patch does not generate ASF License warnings. |
| | | 54m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/445 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile |
| uname | Linux 5a509883dafb 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-445/src/.yetus/personality.sh |
| git revision | master / 4aa5728434378cf164b3677bebac64ba87436ba2 |
| Default Java | Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/testReport/ |
| Max. process+thread count | 1125 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/console |
| versions | git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0 |
| Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "tez-yetus (via GitHub)" <gi...@apache.org>.
tez-yetus commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3615482186
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 7s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | checkstyle | 1m 4s | | master passed |
| +1 :green_heart: | javadoc | 0m 45s | | master passed |
| +0 :ok: | spotbugs | 2m 0s | | tez-runtime-library in master has 241 extant spotbugs warnings. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | codespell | 0m 28s | | No new issues. |
| +1 :green_heart: | compile | 0m 25s | | the patch passed |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 6m 0s | | tez-runtime-library in the patch passed. |
| +1 :green_heart: | asflicense | 0m 13s | | The patch does not generate ASF License warnings. |
| | | 28m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/445 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile |
| uname | Linux 91e2ecc61eca 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-445/src/.yetus/personality.sh |
| git revision | master / 263c049a835f9904a45ec88696eca4f4a3021c38 |
| Default Java | Ubuntu-21.0.9+10-Ubuntu-124.04 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/testReport/ |
| Max. process+thread count | 1109 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/console |
| versions | git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0 |
| Powered by | Apache Yetus 0.15.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #445:
URL: https://github.com/apache/tez/pull/445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3577691323
@maheshrajus: thanks for taking care of this, a few questions:
1. what's the issue that's thrown in the problematic case?
2. what was the requested/available memory in that case?
3. is it 100% sure that `* 10` solves the problem in this case when `+ 100L` failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory [tez]
Posted by "maheshrajus (via GitHub)" <gi...@apache.org>.
maheshrajus commented on PR #445:
URL: https://github.com/apache/tez/pull/445#issuecomment-3579604979
@abstractdog
The **RequestSize** size is coming less in test case failure scenario when compare to **InMemThreshold** value.
INFO [Time-limited test] impl.TestSimpleFetchedInputAllocator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(99)) - **jvmMax: 1073741824**
INFO [Time-limited test] impl.TestSimpleFetchedInputAllocator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(107)) - **InMemThreshold: 107374184**
INFO [Time-limited test] impl.SimpleFetchedInputAllocator (SimpleFetchedInputAllocator.java:<init>(117)) - srcName: RequestedMemory=107374184, AssignedMemory=107374184, maxSingleShuffleLimit=107374184
INFO [Time-limited test] impl.TestSimpleFetchedInputAlloc¸ator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(115)) - **RequestSize: 42949776**
here **RequestSize** should be greater than **InMemThreshold**. Then only it will allocate from the DISK. So it is failing in some env only. In any case we should put more **RequestSize** . So currently `* 10` is solving the issue.
Test sample:
```
// check if requestSize is greater than maxSingleShuffleLimit
assertTrue(requestSize > inputManager.maxSingleShuffleLimit);
// requestSize is greater than the maxSingleShuffleLimit, so allocation is from DISK
FetchedInput fi1 = inputManager.allocate(requestSize, compressedSize, new InputAttemptIdentifier(1, 1));
assertEquals(FetchedInput.Type.DISK, fi1.getType());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org