slow writes to HDFS

[Mark Kerzner <ma...@gmail.com>]

Hi all,

I am writing to HDFS with this simple code

            File[] files = new File(fileDir).listFiles();
            for (File file : files) {
                key.set(file.getPath());
                byte[] bytes = new FileUtil().readCompleteFile(file);
                System.out.println(file.getName() + ", size=" +
bytes.length);
                value = new BytesWritable(bytes);
                writer.append(key, value);  // apparently majority of time
taken by this line
            }

and I notice that writing takes about 1 sec per Meg, so the whole directory
takes about a minute, while cp copies them all in under one second.

Am I doing something wrong? I am running on a single fast machine.

Thank you,
Mark