[jira] [Work logged] (HADOOP-16810) Increase entropy to improve cryptographic randomness on precommit Linux VMs

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-16810?focusedWorklogId=552058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-552058 ]

ASF GitHub Bot logged work on HADOOP-16810:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 12/Feb/21 23:29
            Start Date: 12/Feb/21 23:29
    Worklog Time Spent: 10m 
      Work Description: amahussein opened a new pull request #2697:
URL: https://github.com/apache/hadoop/pull/2697


   [HADOOP-16810: Increase entropy to improve cryptographic randomness on precommit Linux VMs](https://issues.apache.org/jira/browse/HADOOP-16810)
   In [my comment on MAPREDUCE-7079](https://issues.apache.org/jira/browse/MAPREDUCE-7079?focusedCommentId=17013234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17013234)
   This test case has been failing for ever.
    - When it timeout, MRAppMaster and some YarnChild processes remain running in the background. Therefore, the JVM running the tests fail due to OOM. No one notices that this unit test case has failed because the QA reports the unit tests that failed, but not timeout.
   - It works for Mac OS X, but never works for Linux running on a virtual Box. It only works on the latter by disabling MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA.
   
   In this PR:
   
   - the `DOCKER_EXTRAARGS` are added to `hadoop.sh` to pass the random mount
   - the version 0.10.0 is not on the release page. So, this is upgrading the Yetus to a released version 0.13.0.
   - adding the mount parameter to start `start-build-env.sh`
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 552058)
    Remaining Estimate: 0h
            Time Spent: 10m

> Increase entropy to improve cryptographic randomness on precommit Linux VMs
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-16810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16810
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ahmed Hussein
>            Priority: Blocker
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> I was investigating a JUnit test (MAPREDUCE-7079 :TestMRIntermediateDataEncryption is failing in precommit builds) that was consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> I found that the test hangs slows or hangs indefinitely whenever Java reads the random file.
> I explored two different ways to get that test case to work properly on my local Linux VM running rel7:
> # To install "haveged" and "rng-tools" on the virtual machine running Rel7. Then, start rngd service {{sudo service rngd start}} . This will fix the problem for all the components on the image including java, native and any other component.
> # Change java configuration to load urandom
> {code:bash}
> sudo vim $JAVA_HOME/jre/lib/security/java.security
> ## Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
> {code}
> The first solution is better because this will fix the problem for everything that requires SSL/TLS or other services that depend upon encryption.
> Since the precommit build runs on Docker, then it would be best to mount {{/dev/urandom}} from the host as {{/dev/random}} into the container:
> {code:java}
> docker run -v /dev/urandom:/dev/random
> {code}
> For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:
> {code:java}
> DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
> {code}
>  ...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org