Migrated Namenode to a new server - ownership changed on some directories

[Jamie Cockrill <ja...@gmail.com>]

Dear HDFS Issues,

Hopefully this is the right place for my query. Today we migrated our
Namenode to a new server as the old server was short on memory. We
stopped the namenode service, copied its meta-data to the new server
and started it on the new server. In between, we changed the hostname
and IP address of the new server to be the same as the old server so
that we wouldn't need to reconfigure all the cluster nodes to point to
the new server.

Interestingly, when the new server booted the namenode service, it
seems to have changed the ownership of a number of directories on the
HDFS. We only noticed this when we came to run some jobs through Oozie
and many of the jobs failed because the directories were now no longer
owned by the Oozie user.

We are now also seeing lots of warnings in the logs such as:

> org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user webuser
> org.apache.hadoop.util.Shell$ExitCodeException: id: webuser: No such user

for this user and a few other users which were present on the old
server and not on the new server. We can trivially re-create the users
on the new server, but I'm not sure why that would cause some of the
directories on HDFS to change ownership and not all of them.

The only possible other explanation may be that when we booted the
Namenode service for the first time, it started initially with the
default blank configuration and not the configuration copied from the
previous server. We stopped it before it got passed the merging of
edits into the fsimage, so we're not entirely sure what the cause
might be.

Any further pointers as to what the cause might be would be gratefully
received. We are using CDH3u3, however I don't think it's an issue
specifically related to Cloudera's distribution.

Thanks,

Jamie