You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/29 14:44:18 UTC
[GitHub] [hadoop] qizhu-lucas opened a new pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
qizhu-lucas opened a new pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833
…erval between batch configurable.
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637557
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -836,6 +836,10 @@
public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600;
public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY = "dfs.datanode.directoryscan.threads";
public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE = "dfs.datanode.reconcile.blocks.batch.size";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT = 1000;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL = "dfs.datanode.reconcile.blocks.batch.interval";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000;
Review comment:
Fixed it in latest PR.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##########
@@ -873,6 +873,22 @@
</description>
</property>
+<property>
+ <name>dfs.datanode.reconcile.blocks.batch.size</name>
+ <value>1000</value>
+ <description>Run reconcile to checkAndUpdate with batch,
Review comment:
Changed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-832482902
Thanks @ayushtkn for patient review.
I have updated in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637716
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -316,6 +317,18 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ reconcileBlocksBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ reconcileBlocksBatchInterval =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
Fixed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-821010516
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 16s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 237m 0s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not generate ASF License warnings. |
| | | 325m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 4b1137f6cfbd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c0eb74eb65b592f02ff289c698a1d0eca546a1d |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/testReport/ |
| Max. process+thread count | 3502 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-813931427
@ayushtkn
Thanks a lot for your review and good suggestions.
I have updated in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r614224919
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -315,6 +316,40 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ int reconcileBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ if (reconcileBatchSize < 0) {
+ LOG.warn(
+ "Invalid value configured for dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+ }
+
+ reconcileBlocksBatchSize = reconcileBatchSize;
+
+ long reconcileBatchInterval =
+ conf.getLong(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
This should use ``conf.getTimeDuration``, If you need some reference can check HDFS-15107, so as how to add support for time units.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##########
@@ -873,6 +873,22 @@
</description>
</property>
+<property>
+ <name>dfs.datanode.reconcile.blocks.batch.size</name>
+ <value>1000</value>
+ <description>HDFS-14476 to enable running reconcile to checkAndUpdate with batch,
Review comment:
No need for jira id in the description, and this doesn't enable/disable, it is just to specify the batch size.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##########
@@ -873,6 +873,22 @@
</description>
</property>
+<property>
+ <name>dfs.datanode.reconcile.blocks.batch.size</name>
+ <value>1000</value>
+ <description>HDFS-14476 to enable running reconcile to checkAndUpdate with batch,
+ setting this to define batch size.
+ </description>
+</property>
+
+<property>
+ <name>dfs.datanode.reconcile.blocks.batch.interval</name>
+ <value>2000</value>
+ <description>Setting this to define interval between
+ batches defined in dfs.datanode.reconcile.blocks.batch.size.
+ </description>
Review comment:
Some grammatical error, Please rectify
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -315,6 +316,40 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ int reconcileBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ if (reconcileBatchSize < 0) {
+ LOG.warn(
+ "Invalid value configured for dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+ }
+
+ reconcileBlocksBatchSize = reconcileBatchSize;
+
+ long reconcileBatchInterval =
+ conf.getLong(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
+
+ if (reconcileBatchInterval < 0) {
+ LOG.warn(
+ "Invalid value configured for dfs.datanode.reconcile.blocks.batch.interval, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchInterval =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT;
+ }
+
+ reconcileBlocksBatchInterval = reconcileBatchInterval;
}
Review comment:
You need to format this code properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809439075
@Hexiaoqiao @ayushtkn
Could you help review this, when you are free?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637557
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -836,6 +836,10 @@
public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600;
public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY = "dfs.datanode.directoryscan.threads";
public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE = "dfs.datanode.reconcile.blocks.batch.size";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT = 1000;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL = "dfs.datanode.reconcile.blocks.batch.interval";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000;
Review comment:
Fixed it in latest patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-814135509
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 13m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 55s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 484 unchanged - 0 fixed = 490 total (was 484) |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 228m 56s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 326m 59s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestBlockRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 0a8593ed3b2a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6350baa0dbc76e2631b5b5918ec180fe45dd64fb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/testReport/ |
| Max. process+thread count | 3500 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-832787902
Thanx @qizhu-lucas for the contribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-832665989
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 22s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 229m 10s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not generate ASF License warnings. |
| | | 317m 14s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 7963828e84a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5b6d90fc419211774a4fc33ae628de33b1ed4f2 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/4/testReport/ |
| Max. process+thread count | 3450 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
ayushtkn merged pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r606714508
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##########
@@ -873,6 +873,22 @@
</description>
</property>
+<property>
+ <name>dfs.datanode.reconcile.blocks.batch.size</name>
+ <value>1000</value>
+ <description>Run reconcile to checkAndUpdate with batch,
Review comment:
can you add some recheck the descriptions for both the configs, the first line in. both is same, and doesn't make sense to me, recheck once.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -316,6 +317,18 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ reconcileBlocksBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ reconcileBlocksBatchInterval =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
Add a validation for these configs, if ``reconcileBlocksBatchSize`` and ``reconcileBlocksBatchInterval`` is less than one use default. and add a warn log message if these values are incorrect something like:
Invalid value configured for < config name>, should be greater than 0, Using default.
In the end add an Info log for the values being used.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -836,6 +836,10 @@
public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600;
public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY = "dfs.datanode.directoryscan.threads";
public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE = "dfs.datanode.reconcile.blocks.batch.size";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT = 1000;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL = "dfs.datanode.reconcile.blocks.batch.interval";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000;
Review comment:
Can we add support for time units?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas edited a comment on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas edited a comment on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809439075
@Hexiaoqiao @ayushtkn @liuml07
Could you help review this, when you are free?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r625692545
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -315,6 +316,41 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ int reconcileBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ if (reconcileBatchSize < 0) {
Review comment:
change it to <=0
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##########
@@ -315,6 +316,41 @@ public DirectoryScanner(FsDatasetSpi<?> dataset, Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+ int reconcileBatchSize =
+ conf.getInt(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+ if (reconcileBatchSize < 0) {
+ LOG.warn("Invalid value configured for " +
+ "dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+ }
+
+ reconcileBlocksBatchSize = reconcileBatchSize;
+
+ long reconcileBatchInterval =
+ conf.getTimeDuration(DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT,
+ TimeUnit.MILLISECONDS);
+
+ if (reconcileBatchInterval < 0) {
Review comment:
change it to <=0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
qizhu-lucas commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-820872414
Thanks very much for @ayushtkn patient review.
I have updated above in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809675715
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 46s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 55s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 484 unchanged - 0 fixed = 488 total (was 484) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 234m 32s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 320m 57s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 62820b092e29 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4bc31c8d1731e7fc88b514488136048d30d365ec |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/testReport/ |
| Max. process+thread count | 3185 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org