setting up a two node hadoop cluster on RHES 64-bit and 32-bit hosts

[Mich Talebzadeh <mi...@peridale.co.uk>]

Hi,

This is a matter of information.

I had a single node Hadoop running on Linux host with RHES5 64-bit and 24GB of RAM.

I also have an old 32-bit Linux host running RHES5 32-bit.

I set up Hadoop on 32-bit box importing directory structure for HADOOP_HOME, Hive and Sqoop from 64-bit host.  Basically tarred the directories on 64-bit host and scped them to 32-bit and untarred them.

Frankly I did not expect Hadoop binaries to work on 32-bit but they did. I just needed to get JDK 32-bit for the 32-bit host

The 32-bit host worked OK. Tested HDFS, YARN, mapred, Hive and Sqoop.

I then decided to create a two-node cluster. I was getting errors first. Then I realised that beside formatting the namenode I had to  go and delete anything under the  storage directories on both nodes. If I recall anything including CURRENT and sub directories.

For Hive, I created the metastore in Sybase ASE as a standalone database. I had to drop and recreate the database as it had old info.

The drawback is obviously one loses the old data. But it worked. In Hadoop log directory for node 1 (the one running namenode) one gets SLOW DISK for mirror warnings. However, I guess that is expected with 100 baseT and 32-bit OS. This was happening when I was importing a 4 million rows table from an Oracle 11g schema (also running on the primary node) via Sqoop and Hive.  Overall it only took an hour to make the two node cluster work.

HTH,

Mich

Sent from my iPad