You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Ryan Garvey <rg...@umbc.edu> on 2012/12/13 15:38:19 UTC
Encrypted Shuffle Help
Hi,
I am relatively new to Hadoop and completely new to SSL encryption. I am
having issues getting encrypted shuffle working on a small test cluster
with Mapreduce V1. I am using self signed certificates I generated with the
java keytool. I followed the instructions on the site Apache Hadoop website
(
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/EncryptedShuffle.html).
The web interfaces for the nodes are using the correct certificates and the
Map phase of jobs work correctly. However reduce tasks always fail.. I've
been using terasort and teragen as test jobs for the cluster. Terasort
always works and teragen fails on the reduce phase every time. I get the
following error in the logs:
2012-12-12 10:10:47,491 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201212120947_0001_r_000000_0 0.0% reduce > copy >
2012-12-12 10:10:47,869 WARN org.mortbay.log: EXCEPTION
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
I generated a key pair for each node in the cluster, Exported each
cert to a trust store and copied the trust store (containing public
keys for every node) to each of the nodes in the cluster. I set both
ssl-server.xml and ssl-client.xml to use the same keystore and
truststore.
I haven't been able to find any documentation on this feature other
than the Apache site and Cloudera's site. Any help would be greatly
appreciated.
Thanks,
Ryan
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Sorry overlooked that you were running Teragen which doesn't execute a
shuffle, hence it's not getting the error. So initial thoughts of issue
being specific to mapreduce program shouldn't be the case.
Regards,
Robert
On Wed, Dec 19, 2012 at 10:28 AM, Robert Molina <rm...@hortonworks.com>wrote:
> Hi Ryan,
> Interesting one mapreduce program works vs. the other which seems to point
> that the error is specific to the mapreduce program. What happens if you
> try the wordcount example?
>
> Regards,
> Robert
>
>
> On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
>
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
>> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Sorry overlooked that you were running Teragen which doesn't execute a
shuffle, hence it's not getting the error. So initial thoughts of issue
being specific to mapreduce program shouldn't be the case.
Regards,
Robert
On Wed, Dec 19, 2012 at 10:28 AM, Robert Molina <rm...@hortonworks.com>wrote:
> Hi Ryan,
> Interesting one mapreduce program works vs. the other which seems to point
> that the error is specific to the mapreduce program. What happens if you
> try the wordcount example?
>
> Regards,
> Robert
>
>
> On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
>
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
>> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Sorry overlooked that you were running Teragen which doesn't execute a
shuffle, hence it's not getting the error. So initial thoughts of issue
being specific to mapreduce program shouldn't be the case.
Regards,
Robert
On Wed, Dec 19, 2012 at 10:28 AM, Robert Molina <rm...@hortonworks.com>wrote:
> Hi Ryan,
> Interesting one mapreduce program works vs. the other which seems to point
> that the error is specific to the mapreduce program. What happens if you
> try the wordcount example?
>
> Regards,
> Robert
>
>
> On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
>
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
>> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Sorry overlooked that you were running Teragen which doesn't execute a
shuffle, hence it's not getting the error. So initial thoughts of issue
being specific to mapreduce program shouldn't be the case.
Regards,
Robert
On Wed, Dec 19, 2012 at 10:28 AM, Robert Molina <rm...@hortonworks.com>wrote:
> Hi Ryan,
> Interesting one mapreduce program works vs. the other which seems to point
> that the error is specific to the mapreduce program. What happens if you
> try the wordcount example?
>
> Regards,
> Robert
>
>
> On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
>
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
>> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
>> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Interesting one mapreduce program works vs. the other which seems to point
that the error is specific to the mapreduce program. What happens if you
try the wordcount example?
Regards,
Robert
On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Interesting one mapreduce program works vs. the other which seems to point
that the error is specific to the mapreduce program. What happens if you
try the wordcount example?
Regards,
Robert
On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Interesting one mapreduce program works vs. the other which seems to point
that the error is specific to the mapreduce program. What happens if you
try the wordcount example?
Regards,
Robert
On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>
Re: Encrypted Shuffle Help
Posted by Robert Molina <rm...@hortonworks.com>.
Hi Ryan,
Interesting one mapreduce program works vs. the other which seems to point
that the error is specific to the mapreduce program. What happens if you
try the wordcount example?
Regards,
Robert
On Thu, Dec 13, 2012 at 6:38 AM, Ryan Garvey <rg...@umbc.edu> wrote:
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1806)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:986)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>