You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/09/04 14:43:05 UTC
[GitHub] [hadoop] Likkey opened a new pull request, #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Likkey opened a new pull request, #4848:
URL: https://github.com/apache/hadoop/pull/4848
…ndition check suite.
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
When the configuration item “dfs.client.mmap.cache.size” is set to a negative number, it will cause /hadoop/bin hdfs dfsadmin -safemode provides all the operation options including enter, leave, get, wait and forceExit are invalid, the terminal returns security mode is null and no exceptions are thrown.
[[HDFS-16653](https://issues.apache.org/jira/browse/HDFS-16653)](https://issues.apache.org/jira/browse/HDFS-16653)
### How was this patch tested?
This patch adds maxEvictableMmapedSize that is "dfs.client.mmap.cache.size" related Precondition check suite error message, and give a clear indication when the configuration is abnormal in order to solve the problem in time and reduce the impact on the safe mode related operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Likkey commented on pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Posted by GitBox <gi...@apache.org>.
Likkey commented on PR #4848:
URL: https://github.com/apache/hadoop/pull/4848#issuecomment-1237061518
> @Likkey Thank you very much for your contribution. Can we provide the stack before and after modification ?
Thank you very much for your reply!!!!
Before the modification the terminal output looked like this:
```
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode leave
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode enter
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode get
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode forceExit
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
```
And after the modification provides a clear reminder that:
```
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode leave
safemode: Invalid argument: dfs.client.mmap.cache.size must be greater than zero.
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4848:
URL: https://github.com/apache/hadoop/pull/4848#issuecomment-1236434098
Can we provide the stack before and after modification ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Likkey closed pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Posted by "Likkey (via GitHub)" <gi...@apache.org>.
Likkey closed pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
URL: https://github.com/apache/hadoop/pull/4848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4848:
URL: https://github.com/apache/hadoop/pull/4848#issuecomment-1236375953
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 11s | | trunk passed |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4848/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. |
| | | 111m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4848/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4848 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 411718126ac4 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b921e6a09b18d9608ece56b18915ad5575ed193 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4848/1/testReport/ |
| Max. process+thread count | 604 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4848/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Likkey commented on pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.
Posted by GitBox <gi...@apache.org>.
Likkey commented on PR #4848:
URL: https://github.com/apache/hadoop/pull/4848#issuecomment-1240156332
> @Likkey Thank you very much for your contribution. Can we provide the stack before and after modification ?
@slfan1989 Thank you very much for your reply!!!!
In fact, the stack information before and after the change is essentially unchanged.
I think that since the constructor of the ShortCircuitCache class in ShortCircuitCache.java in the source code already uses Preconditions.checkArgument() to check whether the configuration item value is greater than or equal to zero.So when set to a negative number, it will lead to the creation of ShortCircuitCache class object in ClientContext.java failed.
https://github.com/apache/hadoop/blob/42c8f61fecd89d63af3f91d06bd5702a7f8791b3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java#L170
But due to Preconditions.checkArgument () in the lack of error information, resulting in the terminal using hdfs dfsadmin script appears as follows：
```
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode leave
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode enter
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode get
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode forceExit
safemode: null
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
```
And hdfs logs and terminal are not related to the exception thrown.
Therefore, the cause of the situation can be found directly after adding an error message to the original Preconditions.checkArgument(), as follows：
```
hadoop@ljq1:~/hadoop-3.1.3-work/sbin$ hdfs dfsadmin -safemode leave
safemode: Invalid argument: dfs.client.mmap.cache.size must be greater than zero.
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait | forceExit]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org