You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ahmed Hussein <a...@ahussein.me> on 2020/01/14 18:42:00 UTC
Needs support to add more entropy to improve cryptographic randomness
on Linux VMs
Hi,
I was investigating a JUnit test
(MAPREDUCE-7079:TestMRIntermediateDataEncryption
is failing in precommit builds
<https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads
the random file.
I explored two different ways to get that test case to work properly on my
local Linux VM running rel7:
1. To install "haveged" and "rng-tools" on the virtual machine running
Rel7. Then, start rngd service {{sudo service rngd start}} .
2. Change java configuration to load urandom
{{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
## Change the line “securerandom.source=file:/dev/random” to read:
securerandom.source=file:/dev/./urandom
Is it possible to apply any of the above solutions to the VM that runs the
precommit builds?
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Thanks for flagging this Ahmed.
urandom is a perfectly acceptable option. I am not sure who maintains the pre-commit machines these days.
> On Jan 14, 2020, at 10:42 AM, Ahmed Hussein <a...@ahussein.me> wrote:
>
> Hi,
>
> I was investigating a JUnit test
> (MAPREDUCE-7079:TestMRIntermediateDataEncryption
> is failing in precommit builds
> <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
> consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> I found that the test hangs slows or hangs indefinitely whenever Java reads
> the random file.
>
> I explored two different ways to get that test case to work properly on my
> local Linux VM running rel7:
>
> 1. To install "haveged" and "rng-tools" on the virtual machine running
> Rel7. Then, start rngd service {{sudo service rngd start}} .
> 2. Change java configuration to load urandom
> {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
> ## Change the line “securerandom.source=file:/dev/random” to read:
> securerandom.source=file:/dev/./urandom
>
>
> Is it possible to apply any of the above solutions to the VM that runs the
> precommit builds?
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Ahmed Hussein <a...@ahussein.me>.
It turned out that INFRA was not the right place to file.
I created another one HADOOP-16810
<https://issues.apache.org/jira/browse/HADOOP-16810> suggesting a quick fix
for docker.
Basically, it requires passing "-v /dev/urandom:/dev/random" to the docker
that runs the pre-commit tests. Any idea how to do that?
On Wed, Jan 15, 2020 at 11:17 AM Ahmed Hussein <a...@ahussein.me> wrote:
> I filed a new Jira INFRA-19730
> <https://issues.apache.org/jira/browse/INFRA-19730>
>
> On Wed, Jan 15, 2020 at 10:58 AM Ahmed Hussein <a...@ahussein.me> wrote:
>
>>
>> For testing, it is important to be able to use Random instead of
>>> SecureRandom. We should never be using the OS entropy for unit tests.
>>>
>>
>> Is this feasible to do? I assume that going that road implies scanning
>> through all objects and classes and override the methods to consume
>> "java.util.Random" instead of "SecureRandom". For example, the JVM may get
>> stuck initializing "KeyGenerator", and changing that to Random seems a
>> painful task.
>>
>> urandom is a perfectly acceptable option. I am not sure who maintains the
>>> pre-commit machines these days.
>>>
>>
>> Another thing to put into consideration is that changing the Java config
>> will only fix the problem for Java, but the Docker image may get stuck with
>> a similar issue in native or SSL.
>> This makes installing the rng service the reliable way to make the docker
>> image stable.
>>
>>
>>
>>> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein <a...@ahussein.me> wrote:
>>>
>>>> I tried setting JVM option "-Djava.security.egd=file:///dev/./urandom"
>>>> from the command line but it did not work for me.
>>>> In my case, I assume the JVM ignores "java.security.egd" because it is
>>>> considered a security thing.
>>>>
>>>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth <fa...@gmail.com> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > based on this article, we might not need infra to do this, but we can
>>>> > specify the /dev/urandom as a random source directly to the JVM for
>>>> test
>>>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>>>> >
>>>> >
>>>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>>>> >
>>>> > Pifta
>>>> >
>>>> > Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020.
>>>> jan.
>>>> > 14., K, 20:14):
>>>> >
>>>> >> You should file an INFRA jira asking about this. They can get in
>>>> touch
>>>> >> with the folks who maintain the Hadoop labeled nodes.
>>>> >>
>>>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me>
>>>> wrote:
>>>> >> >
>>>> >> > Hi,
>>>> >> >
>>>> >> > I was investigating a JUnit test
>>>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>>>> >> > is failing in precommit builds
>>>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>>>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>>>> >> > I found that the test hangs slows or hangs indefinitely whenever
>>>> Java
>>>> >> reads
>>>> >> > the random file.
>>>> >> >
>>>> >> > I explored two different ways to get that test case to work
>>>> properly on
>>>> >> my
>>>> >> > local Linux VM running rel7:
>>>> >> >
>>>> >> > 1. To install "haveged" and "rng-tools" on the virtual machine
>>>> >> running
>>>> >> > Rel7. Then, start rngd service {{sudo service rngd start}} .
>>>> >> > 2. Change java configuration to load urandom
>>>> >> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>>>> >> > ## Change the line “securerandom.source=file:/dev/random” to
>>>> read:
>>>> >> > securerandom.source=file:/dev/./urandom
>>>> >> >
>>>> >> >
>>>> >> > Is it possible to apply any of the above solutions to the VM that
>>>> runs
>>>> >> the
>>>> >> > precommit builds?
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> busbey
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>> >>
>>>> >>
>>>> >
>>>> > --
>>>> > Pifta
>>>> >
>>>>
>>>
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Ahmed Hussein <a...@ahussein.me>.
I filed a new Jira INFRA-19730
<https://issues.apache.org/jira/browse/INFRA-19730>
On Wed, Jan 15, 2020 at 10:58 AM Ahmed Hussein <a...@ahussein.me> wrote:
>
> For testing, it is important to be able to use Random instead of
>> SecureRandom. We should never be using the OS entropy for unit tests.
>>
>
> Is this feasible to do? I assume that going that road implies scanning
> through all objects and classes and override the methods to consume
> "java.util.Random" instead of "SecureRandom". For example, the JVM may get
> stuck initializing "KeyGenerator", and changing that to Random seems a
> painful task.
>
> urandom is a perfectly acceptable option. I am not sure who maintains the
>> pre-commit machines these days.
>>
>
> Another thing to put into consideration is that changing the Java config
> will only fix the problem for Java, but the Docker image may get stuck with
> a similar issue in native or SSL.
> This makes installing the rng service the reliable way to make the docker
> image stable.
>
>
>
>> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein <a...@ahussein.me> wrote:
>>
>>> I tried setting JVM option "-Djava.security.egd=file:///dev/./urandom"
>>> from the command line but it did not work for me.
>>> In my case, I assume the JVM ignores "java.security.egd" because it is
>>> considered a security thing.
>>>
>>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth <fa...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > based on this article, we might not need infra to do this, but we can
>>> > specify the /dev/urandom as a random source directly to the JVM for
>>> test
>>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>>> >
>>> >
>>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>>> >
>>> > Pifta
>>> >
>>> > Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020.
>>> jan.
>>> > 14., K, 20:14):
>>> >
>>> >> You should file an INFRA jira asking about this. They can get in touch
>>> >> with the folks who maintain the Hadoop labeled nodes.
>>> >>
>>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
>>> >> >
>>> >> > Hi,
>>> >> >
>>> >> > I was investigating a JUnit test
>>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>>> >> > is failing in precommit builds
>>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>>> >> > I found that the test hangs slows or hangs indefinitely whenever
>>> Java
>>> >> reads
>>> >> > the random file.
>>> >> >
>>> >> > I explored two different ways to get that test case to work
>>> properly on
>>> >> my
>>> >> > local Linux VM running rel7:
>>> >> >
>>> >> > 1. To install "haveged" and "rng-tools" on the virtual machine
>>> >> running
>>> >> > Rel7. Then, start rngd service {{sudo service rngd start}} .
>>> >> > 2. Change java configuration to load urandom
>>> >> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>>> >> > ## Change the line “securerandom.source=file:/dev/random” to
>>> read:
>>> >> > securerandom.source=file:/dev/./urandom
>>> >> >
>>> >> >
>>> >> > Is it possible to apply any of the above solutions to the VM that
>>> runs
>>> >> the
>>> >> > precommit builds?
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> busbey
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >>
>>> >>
>>> >
>>> > --
>>> > Pifta
>>> >
>>>
>>
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Ahmed Hussein <a...@ahussein.me>.
> For testing, it is important to be able to use Random instead of
> SecureRandom. We should never be using the OS entropy for unit tests.
>
Is this feasible to do? I assume that going that road implies scanning
through all objects and classes and override the methods to consume
"java.util.Random" instead of "SecureRandom". For example, the JVM may get
stuck initializing "KeyGenerator", and changing that to Random seems a
painful task.
urandom is a perfectly acceptable option. I am not sure who maintains the
> pre-commit machines these days.
>
Another thing to put into consideration is that changing the Java config
will only fix the problem for Java, but the Docker image may get stuck with
a similar issue in native or SSL.
This makes installing the rng service the reliable way to make the docker
image stable.
> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein <a...@ahussein.me> wrote:
>
>> I tried setting JVM option "-Djava.security.egd=file:///dev/./urandom"
>> from the command line but it did not work for me.
>> In my case, I assume the JVM ignores "java.security.egd" because it is
>> considered a security thing.
>>
>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth <fa...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > based on this article, we might not need infra to do this, but we can
>> > specify the /dev/urandom as a random source directly to the JVM for test
>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>> >
>> >
>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>> >
>> > Pifta
>> >
>> > Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020. jan.
>> > 14., K, 20:14):
>> >
>> >> You should file an INFRA jira asking about this. They can get in touch
>> >> with the folks who maintain the Hadoop labeled nodes.
>> >>
>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I was investigating a JUnit test
>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>> >> > is failing in precommit builds
>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>> >> > I found that the test hangs slows or hangs indefinitely whenever Java
>> >> reads
>> >> > the random file.
>> >> >
>> >> > I explored two different ways to get that test case to work properly
>> on
>> >> my
>> >> > local Linux VM running rel7:
>> >> >
>> >> > 1. To install "haveged" and "rng-tools" on the virtual machine
>> >> running
>> >> > Rel7. Then, start rngd service {{sudo service rngd start}} .
>> >> > 2. Change java configuration to load urandom
>> >> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>> >> > ## Change the line “securerandom.source=file:/dev/random” to read:
>> >> > securerandom.source=file:/dev/./urandom
>> >> >
>> >> >
>> >> > Is it possible to apply any of the above solutions to the VM that
>> runs
>> >> the
>> >> > precommit builds?
>> >>
>> >>
>> >>
>> >> --
>> >> busbey
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>
>> >>
>> >
>> > --
>> > Pifta
>> >
>>
>
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Owen O'Malley <ow...@gmail.com>.
For testing, it is important to be able to use Random instead of
SecureRandom. We should never be using the OS entropy for unit tests.
.. Owen
On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein <a...@ahussein.me> wrote:
> I tried setting JVM option "-Djava.security.egd=file:///dev/./urandom"
> from the command line but it did not work for me.
> In my case, I assume the JVM ignores "java.security.egd" because it is
> considered a security thing.
>
> On Tue, Jan 14, 2020 at 1:27 PM István Fajth <fa...@gmail.com> wrote:
>
> > Hi,
> >
> > based on this article, we might not need infra to do this, but we can
> > specify the /dev/urandom as a random source directly to the JVM for test
> > runs with the option: -Djava.security.egd=file:///dev/urandom.
> >
> >
> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
> >
> > Pifta
> >
> > Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020. jan.
> > 14., K, 20:14):
> >
> >> You should file an INFRA jira asking about this. They can get in touch
> >> with the folks who maintain the Hadoop labeled nodes.
> >>
> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
> >> >
> >> > Hi,
> >> >
> >> > I was investigating a JUnit test
> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
> >> > is failing in precommit builds
> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> >> > I found that the test hangs slows or hangs indefinitely whenever Java
> >> reads
> >> > the random file.
> >> >
> >> > I explored two different ways to get that test case to work properly
> on
> >> my
> >> > local Linux VM running rel7:
> >> >
> >> > 1. To install "haveged" and "rng-tools" on the virtual machine
> >> running
> >> > Rel7. Then, start rngd service {{sudo service rngd start}} .
> >> > 2. Change java configuration to load urandom
> >> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
> >> > ## Change the line “securerandom.source=file:/dev/random” to read:
> >> > securerandom.source=file:/dev/./urandom
> >> >
> >> >
> >> > Is it possible to apply any of the above solutions to the VM that runs
> >> the
> >> > precommit builds?
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >>
> >
> > --
> > Pifta
> >
>
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Ahmed Hussein <a...@ahussein.me>.
I tried setting JVM option "-Djava.security.egd=file:///dev/./urandom"
from the command line but it did not work for me.
In my case, I assume the JVM ignores "java.security.egd" because it is
considered a security thing.
On Tue, Jan 14, 2020 at 1:27 PM István Fajth <fa...@gmail.com> wrote:
> Hi,
>
> based on this article, we might not need infra to do this, but we can
> specify the /dev/urandom as a random source directly to the JVM for test
> runs with the option: -Djava.security.egd=file:///dev/urandom.
>
> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>
> Pifta
>
> Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020. jan.
> 14., K, 20:14):
>
>> You should file an INFRA jira asking about this. They can get in touch
>> with the folks who maintain the Hadoop labeled nodes.
>>
>> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
>> >
>> > Hi,
>> >
>> > I was investigating a JUnit test
>> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>> > is failing in precommit builds
>> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>> > I found that the test hangs slows or hangs indefinitely whenever Java
>> reads
>> > the random file.
>> >
>> > I explored two different ways to get that test case to work properly on
>> my
>> > local Linux VM running rel7:
>> >
>> > 1. To install "haveged" and "rng-tools" on the virtual machine
>> running
>> > Rel7. Then, start rngd service {{sudo service rngd start}} .
>> > 2. Change java configuration to load urandom
>> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>> > ## Change the line “securerandom.source=file:/dev/random” to read:
>> > securerandom.source=file:/dev/./urandom
>> >
>> >
>> > Is it possible to apply any of the above solutions to the VM that runs
>> the
>> > precommit builds?
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
> --
> Pifta
>
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by István Fajth <fa...@gmail.com>.
Hi,
based on this article, we might not need infra to do this, but we can
specify the /dev/urandom as a random source directly to the JVM for test
runs with the option: -Djava.security.egd=file:///dev/urandom.
https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
Pifta
Sean Busbey <bu...@cloudera.com.invalid> ezt írta (időpont: 2020. jan.
14., K, 20:14):
> You should file an INFRA jira asking about this. They can get in touch
> with the folks who maintain the Hadoop labeled nodes.
>
> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
> >
> > Hi,
> >
> > I was investigating a JUnit test
> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
> > is failing in precommit builds
> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> > I found that the test hangs slows or hangs indefinitely whenever Java
> reads
> > the random file.
> >
> > I explored two different ways to get that test case to work properly on
> my
> > local Linux VM running rel7:
> >
> > 1. To install "haveged" and "rng-tools" on the virtual machine running
> > Rel7. Then, start rngd service {{sudo service rngd start}} .
> > 2. Change java configuration to load urandom
> > {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
> > ## Change the line “securerandom.source=file:/dev/random” to read:
> > securerandom.source=file:/dev/./urandom
> >
> >
> > Is it possible to apply any of the above solutions to the VM that runs
> the
> > precommit builds?
>
>
>
> --
> busbey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
--
Pifta
Re: Needs support to add more entropy to improve cryptographic
randomness on Linux VMs
Posted by Sean Busbey <bu...@cloudera.com.INVALID>.
You should file an INFRA jira asking about this. They can get in touch
with the folks who maintain the Hadoop labeled nodes.
On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein <a...@ahussein.me> wrote:
>
> Hi,
>
> I was investigating a JUnit test
> (MAPREDUCE-7079:TestMRIntermediateDataEncryption
> is failing in precommit builds
> <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
> consistently hanging on Linux VMs and failing Mapreduce pre-builds.
> I found that the test hangs slows or hangs indefinitely whenever Java reads
> the random file.
>
> I explored two different ways to get that test case to work properly on my
> local Linux VM running rel7:
>
> 1. To install "haveged" and "rng-tools" on the virtual machine running
> Rel7. Then, start rngd service {{sudo service rngd start}} .
> 2. Change java configuration to load urandom
> {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
> ## Change the line “securerandom.source=file:/dev/random” to read:
> securerandom.source=file:/dev/./urandom
>
>
> Is it possible to apply any of the above solutions to the VM that runs the
> precommit builds?
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org