Strange datanode requests

[Eric Czech <ec...@gmail.com>]

Hi,

Does anyone know why a datanode might make a request to it's own
dfs.datanode.address port (50010 in my case) every 60 seconds?

Specifically, I'm seeing an error like this every minute:

> tail -f /var/log/hadoop/hdfs/hadoop-hdfs-datanode-myhost.log

2015-05-20 14:07:41,030 ERROR datanode.DataNode (DataXceiver.java:run(253))
- myhost.mycompany.com:50010:DataXceiver error processing unknown operation
 src: /127.0.0.1:39352 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:357)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:177)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:109)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:183)
at java.lang.Thread.run(Thread.java:745)
2015-05-20 14:07:41,144 WARN  mortbay.log (Slf4jLog.java:warn(89)) -
EXCEPTION
javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266)
at
sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631)
at
sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

The error itself is likely related to the fact that we have mutual ssl
authentication enabled but for the life of me, I don't know where to look
in the Hadoop code to see what the *client *is using for its ssl connection
configuration.  I suspect that the client keystore either isn't being set
or that we have to do something special with our ssl certificates to allow
mutual ssl connection to AND from 127.0.0.1 (I already tried setting
hadoop.ssl.hostname.verifier to DEFAULT_AND_LOCALHOST to no avail).

Either way, if somebody knows what DataNode function might involve calling
itself, that would be a huge help!  Thanks.

Hadoop Version = 2.6.0
Java Version = Oracle 1.7.0_67