Memory leak in DFS client

[Gert Pfeifer <pf...@se.inf.tu-dresden.de>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
I am running some code dealing with file system operations (copying
files and deleting). While it is runnung the web interface of the name
node tells me that the heap size grows dramatically.

Are there any server-side data structures that I have to close
explicitly, except FSData{IN|Out}putStreams ? Anything that takes heap
in the name node...

I had something in mind like Statements in JDBC, but I just can't find
anything.

Gert
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQIVAwUBSIQ4Bv4RHiapZN5BAQKEZRAAmqtfLezbX9j3i2PPLGSnywcetiiSKM3D
3CvrzG0gt+MI93GzsVtY/jOZH/m50kA4Ty0gFZfvOLKdbKLF6Z3EUsBFSWaQJUq2
LYBUe187494Tlu4uBVQLAeV0vJaDNiDo1iIjC1nqg8zSy4ucHjYEF1UUH+Y5nfBM
gSPzwXhlM7QF7NRNR3uI8OZ4pbMrSH//mDG16XGUQNFfD/HcBXZZ29ChMbEnUT5z
b2cxxtr5bBNJi5z38VAwfFlIXQa2w5JU/5Sbq48KujMKSaI9uzfAYD+/B1paNfcb
Aqxpdma4cU4BYxBRwkrhrZnA8r1v8t0GvxVu/t0CW08g6r6PUWyfk+W5mqNOs7jl
TCHH0Q74EatUkaXwY8roNAhiOb1nnJFQIZ/OYki2JUdahc0CeqQfy6J0VyAUttMc
o0qahpeXyTREe+XbbCxmks/Q6BP7x0ElLfmXKYuegrvDRgqzPmcPDayYA4vACquJ
Vw8wBA1VfzJid+aBR3M0WxjufK+6+u/JMoPMa6MdU7YFgsesGyUWWti1jjIGx1+C
fgXhTtKPA2eO/shj+svt9Ivn1Zdsi+phD5CsRavRbJCzcBSfP7ByYsqtYX4XfP5J
0STPZFqLESA+or5O2bAL2nwRtIMnYzCg7GQrpIfJAll+YE8LPVlteQCg6BpoLlAS
/tqLmL/DI98=
=ATq2
-----END PGP SIGNATURE-----

Re: Memory leak in DFS client

[Gert Pfeifer <pf...@se.inf.tu-dresden.de>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I found out that it is not a bug in my code. I can run a

bin $ ./hadoop fs -ls /seDNS/data/33
ls: timed out waiting for rpc response

It times out for this directory, but before it does so, the name node
takes 2GB more heap and never gives it back.

Any ideas?

Gert


Gert Pfeifer wrote:
> Hi,
> I am running some code dealing with file system operations (copying
> files and deleting). While it is runnung the web interface of the name
> node tells me that the heap size grows dramatically.
> 
> Are there any server-side data structures that I have to close
> explicitly, except FSData{IN|Out}putStreams ? Anything that takes heap
> in the name node...
> 
> I had something in mind like Statements in JDBC, but I just can't find
> anything.
> 
> Gert
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQIVAwUBSIRKA/4RHiapZN5BAQKosRAA81JlhloItywcgwwuA8kxm/aRLDzzyAs+
EBvaC3FtJvzQuKMo8oxQtYliCxb3xMqi78Bg9DkRHB+xV2rCWVbB0uE8w17CQdLq
HnJ8H9/sz5TkFlLe8kDNBvKCfyMr5LXwVf5CQYIr3vj26pgqt2e49jg2pohuQCaq
g1oF5BzVTBWWGPDMOPjvcl5l1YEfVqZoOT5uZytkYYkvOGonWOrykOoDrDsFt3aH
VkWmY9lvouzsUFeDCeSI7EWrFRMcb7BOf45RhcUOdBJtNKSBLbGj8U5+o5iGB6gk
GY8GVlv27mfH9t0UOPnWAo9SfjIQqxVx95WrZNKFzj0j/XyaX9lyUM5zN055MyrT
ZqDTjWsEq3uWEErKSqvpYY+v5XZJVTa7M7Rb4LSUslhVmEG2+S7UudyjZAZlWmEk
1SPkrnxOUDT/gI/0nS24obCpBmLmM91HtDi88RPGnOVXzp6gcO4oTg46cfXeVCNQ
yCTACKfKzaUaARekPVWt64roM3t7/lbfjc59ZihCUhGwI3pDVs+vaCojyFMHlq0s
TrWUfAdNAdHb7H+d6JcW1SUJ++IL6WQgHigHsq++nOlaEdIHFFaq2z1QTJAfkCcR
D5odrmp4r1PeKxsZSl88yfnSflrvgCc4o4ccG5IVPDIzbMnelBE2NizdTVIKFXdP
0VzISbi9N1Q=
=Hf+i
-----END PGP SIGNATURE-----