java.io.IOException: Stream closed when writen from an Executor-Thread

[Robert Krüger <kr...@signal7.de>]

Hi,

I encountered a strange phenomenon:

I have a benchmark program which tests filesystem performance by
creating and then reading a number of files in a simple loop started
from the main method, which can test normal java.io.File-based FSs and
Hadoop.

I have been using this code for a long time. After we encountered
strange problems on our production system I changed it to run those
operations concurrently using java.util.concurrent.ExecutorService.

Now the following happens:

java.io.IOException: Stream closed.
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.isClosed(DFSClient.java:2235)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2682)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2647)
        at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
        at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
        at com.signal7.core.io.IOUtil.closeQuietly(IOUtil.java:69)
        at com.signal7.hadooptest.HadoopTest$1.run(HadoopTest.java:67)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

The stream has not been closed by me! The same code runs flawlessly on a
normal file system with multithreading and a large number of operations.
However this is reproducible with HDFS even when using a single thread
and a single operation.

Since HDFS was developed with massively parallel operations in mind, I
cannot believe that we're doing something we shouldn't be doing.

Version is 0.18.1 on client and server. Platform is Red Hat Linux.

Any advice anyone? Where can we look for clues? Can this be a
configuration thing? Has anyone had this before?

Thanks in advance,

Robert