You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Klaus Nagel <da...@gibtsdochgar.net> on 2010/01/18 02:44:25 UTC
fuse_dfs and iozone
Hi everybody - new week, new problem...hope you can help me another time...
I want to run iozone in a fuse_dfs mounted directory (command: iozone -a)
but that doesn't work. (hadoop version 0.20.1, System Debian Lenny 32Bit
with 2.6.26-2-686 kernel, only one datanode configured)
###Output from iozone:
Can not open temp file: iozone.tmp
open: Input/output error
###fuse_dfs_wrapper output in debug mode:
LOOKUP /temp/iozone.tmp
unique: 26, error: -2 (No such file or directory), outsize: 16
unique: 27, opcode: CREATE (35), nodeid: 6, insize: 59
NODEID: 7
unique: 27, error: 0 (Success), outsize: 152
unique: 28, opcode: GETATTR (3), nodeid: 6, insize: 56
unique: 28, error: 0 (Success), outsize: 112
unique: 29, opcode: SETATTR (4), nodeid: 7, insize: 128
unique: 29, error: 0 (Success), outsize: 112
unique: 30, opcode: FLUSH (25), nodeid: 7, insize: 64
FLUSH[167051800]
unique: 30, error: 0 (Success), outsize: 16
unique: 31, opcode: RELEASE (18), nodeid: 7, insize: 64
RELEASE[167051800] flags: 0x8001
unique: 32, opcode: UNLINK (10), nodeid: 6, insize: 51
Exception in thread "Thread-6" org.apache.hadoop.ipc.RemoteException:
java.io.IOException: Could not complete write to file /temp/iozone.tmp by
DFSClient_1838156575
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3226)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3150)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
Call to org/apache/hadoop/fs/FSDataOutputStream::close failed!
ERROR: dfs problem - could not close file_handle(166166016) for
/temp/iozone.tmp fuse_impls_release.c:59
unique: 31, error: 0 (Success), outsize: 16
CREATE[167051800] flags: 0x8041 /temp/iozone.tmp
UNLINK /temp/iozone.tmp
unique: 32, error: 0 (Success), outsize: 16
### namenodes log
2010-01-18 02:22:11,535 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file /temp/iozone.tmp is closed by
DFSClient_-585130158
2010-01-18 02:22:11,539 WARN org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: failed to complete /temp/iozone.tmp because
dir.getFileBlocks() is null and pendingFile is null
2010-01-18 02:22:11,539 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 9000, call complete(/temp/iozone.tmp, DFSClient_-585130158)
from 10.8.0.1:49371: error: java.io.IOException: Could not complete write
to file /temp/iozone.tmp by DFSClient_-585130158
java.io.IOException: Could not complete write to file /temp/iozone.tmp by
DFSClient_-585130158
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2010-01-18 02:22:11,590 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop,fuse ip=/10.8.0.1 cmd=delete
src=/temp/iozone.tmp dst=null perm=null
### nothing ist logged about that operation in the datanode
...that was quiet all - I tested it with many different iozone options,
but everytime the same result...(making a file with e.g. echo ... > file
works perfectly)
...hope someone knows about that problem...
thanks for any reply: klaus
Re: fuse_dfs and iozone
Posted by Eli Collins <el...@cloudera.com>.
On Sun, Jan 17, 2010 at 5:44 PM, Klaus Nagel <da...@gibtsdochgar.net> wrote:
> Hi everybody - new week, new problem...hope you can help me another time...
>
> I want to run iozone in a fuse_dfs mounted directory (command: iozone -a)
> but that doesn't work. (hadoop version 0.20.1, System Debian Lenny 32Bit
> with 2.6.26-2-686 kernel, only one datanode configured)
>
>
> ###Output from iozone:
> Can not open temp file: iozone.tmp
> open: Input/output error
>
>
> ###fuse_dfs_wrapper output in debug mode:
> LOOKUP /temp/iozone.tmp
> unique: 26, error: -2 (No such file or directory), outsize: 16
> unique: 27, opcode: CREATE (35), nodeid: 6, insize: 59
> NODEID: 7
> unique: 27, error: 0 (Success), outsize: 152
> unique: 28, opcode: GETATTR (3), nodeid: 6, insize: 56
> unique: 28, error: 0 (Success), outsize: 112
> unique: 29, opcode: SETATTR (4), nodeid: 7, insize: 128
> unique: 29, error: 0 (Success), outsize: 112
> unique: 30, opcode: FLUSH (25), nodeid: 7, insize: 64
> FLUSH[167051800]
> unique: 30, error: 0 (Success), outsize: 16
> unique: 31, opcode: RELEASE (18), nodeid: 7, insize: 64
> RELEASE[167051800] flags: 0x8001
> unique: 32, opcode: UNLINK (10), nodeid: 6, insize: 51
> Exception in thread "Thread-6" org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: Could not complete write to file /temp/iozone.tmp by
> DFSClient_1838156575
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.complete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.complete(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3226)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3150)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
> Call to org/apache/hadoop/fs/FSDataOutputStream::close failed!
> ERROR: dfs problem - could not close file_handle(166166016) for
> /temp/iozone.tmp fuse_impls_release.c:59
> unique: 31, error: 0 (Success), outsize: 16
> CREATE[167051800] flags: 0x8041 /temp/iozone.tmp
> UNLINK /temp/iozone.tmp
> unique: 32, error: 0 (Success), outsize: 16
>
>
> ### namenodes log
> 2010-01-18 02:22:11,535 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file /temp/iozone.tmp is closed by
> DFSClient_-585130158
> 2010-01-18 02:22:11,539 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: failed to complete /temp/iozone.tmp because
> dir.getFileBlocks() is null and pendingFile is null
> 2010-01-18 02:22:11,539 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 9000, call complete(/temp/iozone.tmp, DFSClient_-585130158)
> from 10.8.0.1:49371: error: java.io.IOException: Could not complete write
> to file /temp/iozone.tmp by DFSClient_-585130158
> java.io.IOException: Could not complete write to file /temp/iozone.tmp by
> DFSClient_-585130158
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> 2010-01-18 02:22:11,590 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
> ugi=hadoop,hadoop,fuse ip=/10.8.0.1 cmd=delete
> src=/temp/iozone.tmp dst=null perm=null
>
>
> ### nothing ist logged about that operation in the datanode
>
> ...that was quiet all - I tested it with many different iozone options,
> but everytime the same result...(making a file with e.g. echo ... > file
> works perfectly)
>
> ...hope someone knows about that problem...
>
> thanks for any reply: klaus
>
>
Looks like the file is being unlinked while the client is still
completing the creation of the file. Would need to see all the
fuse_dfs_wrapper output and what iozone was doing to really understand
what's going on.
Thanks,
Eli