You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2023/06/09 16:18:33 UTC
[GitHub] [hadoop] steveloughran opened a new pull request, #5728: MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
steveloughran opened a new pull request, #5728:
URL: https://github.com/apache/hadoop/pull/5728
This modifies the manifest committer so that the list of files to rename is passed between stages as a file of writeable entries on the local filesystem.
The map of directories to create is still passed in memory; this map is built across all tasks, so even if many tasks created files, if they all write into the same set of directories the memory needed is O(directories) with the
task count not a factor.
The _SUCCESS file reports on heap size through gauges. This should give a warning if there are problems.
Contributed by Steve Loughran
### How was this patch tested?
Itests in progress; yetus gets to test mapreduce
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5728: MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5728:
URL: https://github.com/apache/hadoop/pull/5728#issuecomment-1585066047
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 17m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 2s | | branch-3.3 passed |
| +1 :green_heart: | compile | 20m 28s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 56s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 4m 57s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 33s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 7m 25s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 27m 2s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 15s | | the patch passed |
| +1 :green_heart: | compile | 19m 56s | | the patch passed |
| +1 :green_heart: | javac | 19m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 2m 44s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5728/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 32 unchanged - 1 fixed = 34 total (was 33) |
| +1 :green_heart: | mvnsite | 4m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 15s | | the patch passed |
| +1 :green_heart: | spotbugs | 7m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 8m 8s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 3m 2s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not generate ASF License warnings. |
| | | 219m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5728/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5728 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 6f61f2e4ee32 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 2925d9f3c81a662930e51a4c80c2268bafdfe51b |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5728/1/testReport/ |
| Max. process+thread count | 2159 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5728/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5728: MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5728:
URL: https://github.com/apache/hadoop/pull/5728#issuecomment-1587268209
stored test against azure cardiff, timeouts in ITestAzureBlobFileSystemE2E but nothing else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5728: MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran merged PR #5728:
URL: https://github.com/apache/hadoop/pull/5728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org