You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/08/13 12:02:13 UTC
[GitHub] [hadoop] jianghuazhu opened a new pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jianghuazhu opened a new pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900063826
Thanks @virajjasani for the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901709391
hmm,
Not sure why the Test Result Link is also not working for most PRs:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/testReport/
@jojochuang are you aware of some thread/jira disabling it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
ferhui commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900750019
@jianghuazhu Thanks for contribution! @virajjasani Thanks for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-898423320
There are some exceptions in CI/CD:
1.org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
2.java.nio.file.FileSystemException: /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-3302: Read-only file system
It seems to be related to the jenkins environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang merged pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900997200
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 35s | | trunk passed |
| -1 :x: | compile | 17m 52s | [/branch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/artifact/out/branch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in trunk failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| +1 :green_heart: | compile | 18m 30s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 20m 33s | [/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 200 new + 1720 unchanged - 0 fixed = 1920 total (was 1720) |
| +1 :green_heart: | compile | 18m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 28s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 17m 6s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 176m 25s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.cli.TestCLI |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux f683d58fe1e6 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9a4658a6c0b3abc8eae85dd213a2ee78f0a609d9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905321805
@ayushtkn , thanks for your help.
I don't know if this PR request can be merged into the trunk branch. If can, can you help me deal with it?
thank you very much. (I'm not sure if I should make such request.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu edited a comment on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu edited a comment on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906256247
@jojochuang , I'm very sorry, I previously overlooked a hidden detail in testConf.xml. I have solved it now.
Can you help review again.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690948417
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +335,22 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ threadPoolQueueSize = 1024;
Review comment:
Thanks @ayushtkn for the comment.
I will make some changes later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r695701899
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +340,25 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ LOG.warn("The value of the thread pool queue size cannot be " +
+ "less than 1, and the default value is used here. " +
+ "The default size is 1024.");
+ threadPoolQueueSize = 1024;
+ } else {
+ threadPoolQueueSize = parsedValue;
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public int getThreadPoolQueueSize() {
Review comment:
This method can be package-private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690938256
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +335,22 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ threadPoolQueueSize = 1024;
Review comment:
Better to add a warn log line here, to denote the thread pool queue size was misconfigured and we are now using the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r696235839
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +340,25 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ LOG.warn("The value of the thread pool queue size cannot be " +
+ "less than 1, and the default value is used here. " +
+ "The default size is 1024.");
+ threadPoolQueueSize = 1024;
+ } else {
+ threadPoolQueueSize = parsedValue;
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public int getThreadPoolQueueSize() {
Review comment:
Thanks @aajisaka for the suggestion, I will continue to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690875212
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
##########
@@ -542,6 +542,8 @@ Options:
* `-l` : Allow DataNode to lazily persist the file to disk, Forces a replication
factor of 1. This flag will result in reduced durability. Use with care.
* `-d` : Skip creation of temporary file with the suffix `._COPYING_`.
+* `-q <threadPool queue size>` : Number of threadPool queue size to be used,
Review comment:
Thanks to @jojochuang comment, I will make some changes later.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -256,6 +258,8 @@ protected void processOptions(LinkedList<String> args)
" -p : Preserves timestamps, ownership and the mode.\n" +
" -f : Overwrites the destination if it already exists.\n" +
" -t <thread count> : Number of threads to be used, default is 1.\n" +
+ " -q <threadPool size> : Number of threadPool queue size to be used, " +
Review comment:
Thanks to @jojochuang comment, I will make some changes later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906900261
+1 we're good to go. Thanks @jianghuazhu !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690872102
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
##########
@@ -542,6 +542,8 @@ Options:
* `-l` : Allow DataNode to lazily persist the file to disk, Forces a replication
factor of 1. This flag will result in reduced durability. Use with care.
* `-d` : Skip creation of temporary file with the suffix `._COPYING_`.
+* `-q <threadPool queue size>` : Number of threadPool queue size to be used,
Review comment:
here, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690875212
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
##########
@@ -542,6 +542,8 @@ Options:
* `-l` : Allow DataNode to lazily persist the file to disk, Forces a replication
factor of 1. This flag will result in reduced durability. Use with care.
* `-d` : Skip creation of temporary file with the suffix `._COPYING_`.
+* `-q <threadPool queue size>` : Number of threadPool queue size to be used,
Review comment:
Thanks to @jojochuang comment, I will make some changes later.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -256,6 +258,8 @@ protected void processOptions(LinkedList<String> args)
" -p : Preserves timestamps, ownership and the mode.\n" +
" -f : Overwrites the destination if it already exists.\n" +
" -t <thread count> : Number of threads to be used, default is 1.\n" +
+ " -q <threadPool size> : Number of threadPool queue size to be used, " +
Review comment:
Thanks to @jojochuang comment, I will make some changes later.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +335,22 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ threadPoolQueueSize = 1024;
Review comment:
Thanks @ayushtkn for the comment.
I will make some changes later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900890305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906245391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 58s | | trunk passed |
| +1 :green_heart: | compile | 22m 14s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 18s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 21m 28s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 28s | | the patch passed |
| +1 :green_heart: | compile | 19m 0s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 16s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 181m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint xml |
| uname | Linux 790bd14567a1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 80bb9d8160cea33005fd38aef75d4bb80eb52bc1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/7/testReport/ |
| Max. process+thread count | 1282 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905339585
@jianghuazhu could you help verify TestCLI isn't broken by this change? We'll have to verify it manually. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
ferhui commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900750019
@jianghuazhu Thanks for contribution! @virajjasani Thanks for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900014653
@ayushtkn , thanks for your comment. I will continue to improve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901643513
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 53m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 3s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 3s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 6s | | trunk passed |
| +1 :green_heart: | compile | 30m 31s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 23s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 23m 5s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 5s | | the patch passed |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 20m 29s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 263m 57s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.cli.TestCLI |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 673e639c01ad 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7abf2a2b408e129c0c0c577bc6cf7aa18e18cf54 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/testReport/ |
| Max. process+thread count | 2282 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-899933088
@jojochuang, @virajjasani, can you help review the code.
If there are some imperfections, welcome to communicate, and I will work hard to promote improvement.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-900890305
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 54s | | trunk passed |
| +1 :green_heart: | compile | 28m 23s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 53s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 27m 51s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 27m 51s | | the patch passed |
| +1 :green_heart: | compile | 24m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 24m 22s | | the patch passed |
| -1 :x: | blanks | 0m 1s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/4/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 17m 44s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not generate ASF License warnings. |
| | | 212m 53s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.cli.TestCLI |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux dd64c000cc47 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c4387749221e1d1c5259acad51a9b217f4321a36 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/4/testReport/ |
| Max. process+thread count | 3125 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901702371
Some exceptions occurred here, related to UT:
hadoop.cli.TestCLI
But it has nothing to do with the code I submitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905351084
Thanks @jojochuang for the comment. I will continue to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690871929
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -256,6 +258,8 @@ protected void processOptions(LinkedList<String> args)
" -p : Preserves timestamps, ownership and the mode.\n" +
" -f : Overwrites the destination if it already exists.\n" +
" -t <thread count> : Number of threads to be used, default is 1.\n" +
+ " -q <threadPool size> : Number of threadPool queue size to be used, " +
Review comment:
Just a nit: remove "Number of"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-899160009
@aajisaka , can you help review the code I submitted.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jianghuazhu commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906256247
@jojochuang , I'm very sorry, I previously overlooked a hidden detail in testConf.xml. Now I have solved him.
Can you help review again.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690938256
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -329,6 +335,22 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+ private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+ int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+ if (parsedValue < 1) {
+ threadPoolQueueSize = 1024;
Review comment:
Better to add a warn log line here, to denote the thread pool queue size was misconfigured and we are now using the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r690871929
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##########
@@ -256,6 +258,8 @@ protected void processOptions(LinkedList<String> args)
" -p : Preserves timestamps, ownership and the mode.\n" +
" -f : Overwrites the destination if it already exists.\n" +
" -t <thread count> : Number of threads to be used, default is 1.\n" +
+ " -q <threadPool size> : Number of threadPool queue size to be used, " +
Review comment:
Just a nit: remove "Number of"
##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
##########
@@ -542,6 +542,8 @@ Options:
* `-l` : Allow DataNode to lazily persist the file to disk, Forces a replication
factor of 1. This flag will result in reduced durability. Use with care.
* `-d` : Skip creation of temporary file with the suffix `._COPYING_`.
+* `-q <threadPool queue size>` : Number of threadPool queue size to be used,
Review comment:
here, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org