You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ahmed Hussein (Jira)" <ji...@apache.org> on 2020/01/17 16:00:00 UTC
[jira] [Updated] (HADOOP-16810) Increase entropy to improve
cryptographic randomness on precommit Linux VMs
[ https://issues.apache.org/jira/browse/HADOOP-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Hussein updated HADOOP-16810:
-----------------------------------
Description:
I was investigating a JUnit test (MAPREDUCE-7079 :TestMRIntermediateDataEncryption is failing in precommit builds) that was consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads the random file.
I explored two different ways to get that test case to work properly on my local Linux VM running rel7:
# The haveged service seeds a system's random source by executing a loop repeatedly and using the differences in the processor's time stamp counter. It ensures entropy never drops below 1000.
# To install "haveged" and "rng-tools" on the virtual machine running Rel7. Then, start rngd service {{sudo service rngd start}} . This will fix the problem for all the components on the image including java, native and any other component.
# Change java configuration to load urandom
{code:bash}
sudo vim $JAVA_HOME/jre/lib/security/java.security
## Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
{code}
The first solution is better because this will fix the problem for everything that requires SSL/TLS or other services that depend upon encryption.
Since the precommit build runs on Docker, then it would be best to mount {{/dev/urandom}} from the host as {{/dev/random}} into the container:
{code:java}
docker run -v /dev/urandom:/dev/random
{code}
For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:
{code:java}
DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
{code}
...
was:
I was investigating a JUnit test (MAPREDUCE-7079 :TestMRIntermediateDataEncryption is failing in precommit builds) that was consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads the random file.
I explored two different ways to get that test case to work properly on my local Linux VM running rel7:
# To install "haveged" and "rng-tools" on the virtual machine running Rel7. Then, start rngd service {{sudo service rngd start}} . This will fix the problem for all the components on the image including java, native and any other component.
# Change java configuration to load urandom
{code:bash}
sudo vim $JAVA_HOME/jre/lib/security/java.security
## Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
{code}
The first solution is better because this will fix the problem for everything that requires SSL/TLS or other services that depend upon encryption.
Since the precommit build runs on Docker, then it would be best to mount {{/dev/urandom}} from the host as {{/dev/random}} into the container:
{code:java}
docker run -v /dev/urandom:/dev/random
{code}
For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:
{code:java}
DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
{code}
...
> Increase entropy to improve cryptographic randomness on precommit Linux VMs
> ---------------------------------------------------------------------------
>
> Key: HADOOP-16810
> URL: https://issues.apache.org/jira/browse/HADOOP-16810
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Ahmed Hussein
> Assignee: Allen Wittenauer
> Priority: Major
>
> I was investigating a JUnit test (MAPREDUCE-7079 :TestMRIntermediateDataEncryption is failing in precommit builds) that was consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> I found that the test hangs slows or hangs indefinitely whenever Java reads the random file.
> I explored two different ways to get that test case to work properly on my local Linux VM running rel7:
> # The haveged service seeds a system's random source by executing a loop repeatedly and using the differences in the processor's time stamp counter. It ensures entropy never drops below 1000.
> # To install "haveged" and "rng-tools" on the virtual machine running Rel7. Then, start rngd service {{sudo service rngd start}} . This will fix the problem for all the components on the image including java, native and any other component.
> # Change java configuration to load urandom
> {code:bash}
> sudo vim $JAVA_HOME/jre/lib/security/java.security
> ## Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
> {code}
> The first solution is better because this will fix the problem for everything that requires SSL/TLS or other services that depend upon encryption.
> Since the precommit build runs on Docker, then it would be best to mount {{/dev/urandom}} from the host as {{/dev/random}} into the container:
> {code:java}
> docker run -v /dev/urandom:/dev/random
> {code}
> For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:
> {code:java}
> DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
> {code}
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org