You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2024/09/18 06:02:19 UTC
[PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
ayushtkn opened a new pull request, #7050:
URL: https://github.com/apache/hadoop/pull/7050
### Description of PR
Fix Hadoop CLI MiniCluster
### How was this patch tested?
Tested Manually
1. **Log on Console**
```
2024-09-18 11:00:50,989 INFO mapreduce.MiniHadoopClusterManager: Started MiniDFSCluster -- namenode on port 62569
```
2. **Tried CLI commands on the cluster**
```
ayushsaxena@ayushsaxena hadoop-3.5.0-SNAPSHOT % bin/hdfs dfs -mkdir hdfs://127.0.0.1:62569/ayush
2024-09-18 11:01:55,405 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ayushsaxena@ayushsaxena hadoop-3.5.0-SNAPSHOT % bin/hdfs dfs -ls hdfs://127.0.0.1:62569/
2024-09-18 11:01:58,113 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
drwxrwxrwx - ayushsaxena supergroup 0 2024-09-18 11:00 hdfs://127.0.0.1:62569/Users
drwxr-xr-x - ayushsaxena supergroup 0 2024-09-18 11:01 hdfs://127.0.0.1:62569/ayush
drwxr-xr-x - ayushsaxena supergroup 0 2024-09-18 11:00 hdfs://127.0.0.1:62569/user
ayushsaxena@ayushsaxena hadoop-3.5.0-SNAPSHOT % bin/hdfs dfs -copyFromLocal bin/hdfs hdfs://127.0.0.1:62569/ayush
2024-09-18 11:02:16,333 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ayushsaxena@ayushsaxena hadoop-3.5.0-SNAPSHOT % bin/hdfs dfs -ls hdfs://127.0.0.1:62569/ayush
2024-09-18 11:02:28,387 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 3 ayushsaxena supergroup 11730 2024-09-18 11:02 hdfs://127.0.0.1:62569/ayush/hdfs
```
3. **Namenode UI**
<img width="1442" alt="image" src="https://github.com/user-attachments/assets/5e5831a5-4f09-4fb5-8439-ffb43c499288">
4. **Cat the config dump file**
```
ayushsaxena@ayushsaxena incubator % cat abc.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
<property><name>mapreduce.jobhistory.jhist.format</name><value>binary</value><final>false</final><source>mapred-default.xml</source></property>
<property><name>dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy.balanced-space-tolerance</name><value>5</value><final>false</final><source>hdfs-default.xml</source></property>
<property><name>fs.s3a.retry.interval</name><value>500ms</value><final>false</final><source>core-default.xml</source></property>
<property><name>dfs.block.access.token.lifetime</name><value>600</value><final>false</final><source>hdfs-default.xml</source></property>
<property><name>mapreduce.job.heap.memory-mb.ratio</name><value>0.8</value><final>false</final><source>mapred-default.xml</source></property>
<property><name>mapreduce.map.log.level</name><value>INFO</value><final>false</final><source>mapred-default.xml</source></property>
<property><name>dfs.namenode.lazypersist.file.scrub.interval.sec</name><value>300</value><final>false</final><source>hdfs-default.xml</source></property>
<property><name>file.bytes-per-checksum</name><value>512</value><final>false</final><source>core-default.xml</source></property>
<property><name>mapreduce.client.completion.pollinterval</name><value>5000</value><final>false</final><source>mapred-default.xml</source></property>
```
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7050:
URL: https://github.com/apache/hadoop/pull/7050#issuecomment-2361393625
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 17m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 39s | | trunk passed |
| +1 :green_heart: | compile | 17m 36s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 16m 17s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 4m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 40s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 40s | | the patch passed |
| +1 :green_heart: | compile | 17m 4s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 17m 4s | | the patch passed |
| +1 :green_heart: | compile | 16m 19s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 16m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 44s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 220m 29s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 127m 19s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 1m 24s | | The patch does not generate ASF License warnings. |
| | | 613m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7050 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux cbf626ae8383 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ea08cae5af8c78f78ae6b88f1166e0d42eccd9ab |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/4/testReport/ |
| Max. process+thread count | 3733 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "vinayakumarb (via GitHub)" <gi...@apache.org>.
vinayakumarb commented on code in PR #7050:
URL: https://github.com/apache/hadoop/pull/7050#discussion_r1769464746
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapterMockitoUtil.java:
##########
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.namenode;
+
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.mockito.ArgumentMatcher;
+import org.mockito.ArgumentMatchers;
+import org.mockito.stubbing.Answer;
+
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil;
+import org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer;
+import org.apache.hadoop.test.Whitebox;
+
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+/**
+ * This is a Mockito based utility class to expose NameNode functionality for unit tests.
+ */
+public class NameNodeAdapterMockitoUtil {
+
+ public static BlockManager spyOnBlockManager(NameNode nn) {
+ BlockManager bmSpy = spy(nn.getNamesystem().getBlockManager());
+ nn.getNamesystem().setBlockManagerForTesting(bmSpy);
+ return bmSpy;
+ }
+
+ public static ReentrantReadWriteLock spyOnFsLock(FSNamesystem fsn) {
+ ReentrantReadWriteLock spy = spy(fsn.getFsLockForTests());
+ fsn.setFsLockForTests(spy);
+ return spy;
+ }
+
+ public static FSImage spyOnFsImage(NameNode nn1) {
+ FSNamesystem fsn = nn1.getNamesystem();
+ FSImage spy = spy(fsn.getFSImage());
+ Whitebox.setInternalState(fsn, "fsImage", spy);
+ return spy;
+ }
+
+ public static JournalSet spyOnJournalSet(NameNode nn) {
+ FSEditLog editLog = nn.getFSImage().getEditLog();
+ JournalSet js = spy(editLog.getJournalSet());
+ editLog.setJournalSetForTesting(js);
+ return js;
+ }
+
+ public static FSNamesystem spyOnNamesystem(NameNode nn) {
+ FSNamesystem fsnSpy = spy(nn.getNamesystem());
+ FSNamesystem fsnOld = nn.namesystem;
+ fsnOld.writeLock();
+ fsnSpy.writeLock();
+ nn.namesystem = fsnSpy;
+ try {
+ FieldUtils.writeDeclaredField(nn.getRpcServer(), "namesystem", fsnSpy, true);
+ FieldUtils.writeDeclaredField(
+ fsnSpy.getBlockManager(), "namesystem", fsnSpy, true);
+ FieldUtils.writeDeclaredField(
+ fsnSpy.getLeaseManager(), "fsnamesystem", fsnSpy, true);
+ FieldUtils.writeDeclaredField(
+ fsnSpy.getBlockManager().getDatanodeManager(),
+ "namesystem", fsnSpy, true);
+ FieldUtils.writeDeclaredField(
+ BlockManagerTestUtil.getHeartbeatManager(fsnSpy.getBlockManager()),
+ "namesystem", fsnSpy, true);
+ } catch (IllegalAccessException e) {
+ throw new RuntimeException("Cannot set spy FSNamesystem", e);
+ } finally {
+ fsnSpy.writeUnlock();
+ fsnOld.writeUnlock();
+ }
+ return fsnSpy;
+ }
+
+ public static FSEditLog spyOnEditLog(NameNode nn) {
+ FSEditLog spyEditLog = spy(nn.getNamesystem().getFSImage().getEditLog());
+ DFSTestUtil.setEditLogForTesting(nn.getNamesystem(), spyEditLog);
+ EditLogTailer tailer = nn.getNamesystem().getEditLogTailer();
+ if (tailer != null) {
+ tailer.setEditLog(spyEditLog);
+ }
+ return spyEditLog;
+ }
+
+ /**
Review Comment:
formatting needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #7050:
URL: https://github.com/apache/hadoop/pull/7050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7050:
URL: https://github.com/apache/hadoop/pull/7050#issuecomment-2365231102
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 12s | | trunk passed |
| +1 :green_heart: | compile | 18m 15s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 16m 25s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 4m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 31s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 40s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 16m 58s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 16m 58s | | the patch passed |
| +1 :green_heart: | compile | 16m 34s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 16m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 36s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 42s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 221m 48s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 126m 57s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not generate ASF License warnings. |
| | | 613m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7050 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 82e87a7271f7 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed19d62d44ad86484a737c59d3a6cf0ac944b9c1 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/5/testReport/ |
| Max. process+thread count | 3313 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7050:
URL: https://github.com/apache/hadoop/pull/7050#issuecomment-2359444996
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 42s | | trunk passed |
| +1 :green_heart: | compile | 17m 23s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 16m 28s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 39s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 16m 55s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 16m 55s | | the patch passed |
| +1 :green_heart: | compile | 19m 0s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 19m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 5m 13s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 265 unchanged - 0 fixed = 266 total (was 265) |
| +1 :green_heart: | mvnsite | 4m 27s | | the patch passed |
| -1 :x: | javadoc | 0m 50s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| +1 :green_heart: | javadoc | 3m 48s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 9m 20s | | the patch passed |
| -1 :x: | shadedclient | 2m 49s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 30s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | unit | 37m 36s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 42s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| +0 :ok: | asflicense | 0m 43s | | ASF License check generated no output? |
| | | 265m 45s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestStorageBlockPoolUsageStdDev |
| | hadoop.hdfs.server.diskbalancer.TestPlanner |
| | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.server.blockmanagement.TestBlockPlacementPolicyDebugLoggingBuilder |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy |
| | hadoop.hdfs.server.diskbalancer.TestConnectors |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.datanode.TestDataNodeFSDataSetSink |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
| | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7050 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 55fc8578563b 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eed3706f3d61ad21bb98ce968519f8b1a21e0994 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/testReport/ |
| Max. process+thread count | 961 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19164. Hadoop CLI MiniCluster is broken [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7050:
URL: https://github.com/apache/hadoop/pull/7050#issuecomment-2358949299
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 32s | | trunk passed |
| +1 :green_heart: | compile | 19m 42s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 18m 31s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 4m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 19m 7s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 19m 7s | | the patch passed |
| +1 :green_heart: | compile | 18m 22s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 18m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 5m 15s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 5 new + 265 unchanged - 0 fixed = 270 total (was 265) |
| +1 :green_heart: | mvnsite | 4m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 7s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 3m 22s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 7m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 46s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 227m 57s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 128m 29s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 637m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7050 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 4754faf02da5 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7d1c21cf7747ec1dec9a1d5e59d13187c9a2a582 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/1/testReport/ |
| Max. process+thread count | 4231 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7050/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org