Write to a remote HDFS cluster on a different LAN

[Manuel Polonio <ma...@bitmonlab.com>]

Hello,

I've got a problem while trying to write events to a remote HDFS cluster
that is on a different LAN, because when Flume is informed of the datanodes
to write to, it receives local (local on the remote cluster) IPs. I've
configured HDFS to use hostnames when informing client and datanodes of
datanodes, by setting next parameters on hdfs-site.xml:

   <property>
      <name>dfs.client.use.datanode.hostname</name>
      <value>true</value>
   </property>

   <property>
      <name>dfs.datanode.use.datanode.hostname</name>
      <value>true</value>
   </property>

I've also properly configured /etc/hosts in all datanodes and client. I've
sniffed nodename->client traffic and it seems that IPs and also hostnames
are being reported, but HDFS sink still log (timeout) exceptions with the
local IPs of the remote LAN.

I think this problem should be very usual and an easy workaround has to
exist.

Any help would be highly appreciated.

Best regards,
Manuel Polonio