namenode ip change and empty file at hdfs becoming non-empty

[fancy dou <fa...@gmail.com>]

Hi All,

We're using hadoop-2.0.0-cdh4.3.2. Recently we relocated our namenode by
shutting it down, changing its ip address and restarting it. During that
process, the hdfs clients were still trying to write to hdfs.

Some clients failed due to an java.util.ConcurrentModificationException
throw out from the LeaseRenewer. Looks like it is the problem addressed by
the https://issues.apache.org/jira/browse/HDFS-5028.

The other clients didn't complain anything. However some of the files they
wrote at hdfs has 0 size although they did write something into the
DFSOutputstream. To make it clear, when the client writes file, it firstly
writes to a temp file at hdfs. Once it's completed, the temp file is rename
under the same directory. Later some time, the file will be renamed again
to be put to a different directory. We found after the namenode relocation,
these clients keeping generating empty files at hdfs sometimes but not all
the time for a few days.

We finally restarted all the clients. No new empty files anymore. But the
strange thing is, gradually some files which were empty before became
none-empty.

Here's my question. There's a bug in our program: we didn't handle the
exception when we close the FSDataOutputstream. So it's reasonable to see
empty files.
1. But why did our clients write empty file after the relocation was done.
There should be no connection issue to hadoop since they can write some
files right. I don't understand why sometimes they cannot.
2. After restarting clients, why do some empty files became non-empty?
3. Why didn't all the clients get the ConcurrentModificationException from
LeaseRenewer? And for those clients that don't have such exceptions, what
happened to their lease, outputstream?

Please let me know if you need more information.

Thanks a lot!

Lei