You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Mark Desnoyer <md...@gmail.com> on 2009/08/04 00:05:49 UTC
KMeans Leaked TCP Connections
Hi,
Has anybody had a problem running KMeans leaking TCP connections, which
eventually causes the OS to issue a "Too Many Files Open" error? In other
words, it looks like a file descriptor leak.
I'm running a four machine cluster of Hadoop 18.3, with the current version
of Mahout from the trunk. Each k-means iteration is causing an extra 40 or
so TCP connections to be created that get orphaned in CLOSE_WAIT. I've
attached the output of lsof from the middle of a run.
Any help would be appreciated!
Cheers,
Mark
Re: KMeans Leaked TCP Connections
Posted by Mark Desnoyer <md...@gmail.com>.
The lsof command line args are (where 19625 was the PID of the java binary):
lsof -p 19625
When running Hadoop, the client java version is:
>./java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode)
The call to KMeans is:
KMeansDriver.runJob(input, initialClusterDir_, output, measureClass_,
convergenceDelta_, maxIterations_, k_, DenseVector.class);
where:
input, initialClusterDir_ and output are all HDFS directories with the
appropriate data in mahout.matrix.Vector or mahout.clustering.kmeans.Cluster
objects in SequenceFileFormat. The input is about 100k entries each of 200
dimensions and the initial clustering is a random subsampling.
measureClass_ = EuclideanDistanceMeasure.class.getName()
convergenceDelta_ = 0.0001
maxIterations_ = 100
k = 50
Finally, an example job configuration file from one of the iterations is:
namevalue *fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem *
dfs.client.buffer.dir*${hadoop.tmp.dir}/dfs/tmp *mapred.task.cache.levels*2
*hadoop.tmp.dir*/tmp/hadoop-${user.name} *hadoop.native.lib*true *
map.sort.class*org.apache.hadoop.util.QuickSort *ipc.client.idlethreshold*
4000 *mapred.system.dir*${hadoop.tmp.dir}/mapred/system *
mapred.job.tracker.persist.jobstatus.hours*0 *dfs.namenode.logging.level*
info *dfs.datanode.address*0.0.0.0:50010 *io.skip.checksum.errors*false *
fs.default.name*hdfs://c17-dw-dev50-hdfs-dn-n1.corn.com:8020/ *
mapred.child.tmp*./tmp *dfs.safemode.threshold.pct*0.999f *
dfs.namenode.handler.count*10 *dfs.blockreport.initialDelay*0 *
mapred.tasktracker.dns.nameserver*default *io.sort.factor*10 *
mapred.task.timeout*1200000 *mapred.max.tracker.failures*4 *
hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory *mapred.mapoutput.key.class*
org.apache.hadoop.io.Text *fs.hdfs.impl*
org.apache.hadoop.dfs.DistributedFileSystem *mapred.output.key.class*
org.apache.hadoop.io.Text *dfs.safemode.extension*30000 *
tasktracker.http.threads*40 *fs.ftp.impl*
org.apache.hadoop.fs.ftp.FTPFileSystem *user.name*mdesnoyer *
mapred.output.compress*false *io.bytes.per.checksum*512 *
topology.node.switch.mapping.impl*org.apache.hadoop.net.ScriptBasedMapping
*mapred.reduce.max.attempts*6 *fs.ramfs.impl*
org.apache.hadoop.fs.InMemoryFileSystem *
mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo *
hadoop.job.ugi*mdesnoyer,mdesnoyer *mapred.jar*
/var/opt/hadoop/c17-dwdev/dfs/dfs0/mapred/local/jobTracker/job_200907171112_0508.jar
*fs.s3.buffer.dir*${hadoop.tmp.dir}/s3 *dfs.block.size*67108864 *
job.end.retry.attempts*0 *org.apache.mahout.clustering.kmeans.measure*
org.apache.mahout.utils.EuclideanDistanceMeasure *fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem *mapred.local.dir.minspacestart*
21474836480 *mapred.output.compression.type*BLOCK *
dfs.datanode.ipc.address*0.0.0.0:50020 *dfs.permissions*true *
topology.script.number.args*20 *mapred.task.profile.maps*0-2 *
dfs.datanode.https.address*0.0.0.0:50475 *mapred.userlog.retain.hours*24 *
dfs.secondary.http.address*0.0.0.0:50090 *dfs.replication.max*512 *
mapred.job.tracker.persist.jobstatus.active*false *local.cache.size*
10737418240 *mapred.min.split.size*0 *mapred.map.tasks*218 *
mapred.child.java.opts*-Xmx1024m *mapred.output.value.class*
org.apache.mahout.clustering.kmeans.Cluster *dfs.https.address*
0.0.0.0:50470 *dfs.balance.bandwidthPerSec*1048576 *
ipc.server.listen.queue.size*128 *mapred.inmem.merge.threshold*1000 *
job.end.retry.interval*30000 *fs.checkpoint.dir*
/var/opt/hadoop/${hadoop.ident}/dfs/dfs0/namesecondary *mapred.reduce.tasks
*50 *mapred.userlog.limit.kb*0 *webinterface.private.actions*false *
dfs.max.objects*0 *io.sort.spill.percent*0.80 *mapred.job.split.file*
hdfs://c17-dw-dev50-hdfs-dn-n1/tmp/hadoop-httpds/mapred/system/job_200907171112_0508/job.split
*mapred.job.name*lda-mapreduce.jar *mapred.map.tasks.speculative.execution*
true *dfs.datanode.dns.nameserver*default *dfs.blockreport.intervalMsec*
3600000 *mapred.map.max.attempts*6 *mapred.job.tracker.handler.count*2 *
dfs.client.block.write.retries*3 *mapred.input.format.class*
org.apache.hadoop.mapred.SequenceFileInputFormat *
mapred.tasktracker.expiry.interval*600000 *keep.failed.task.files*false *
mapred.output.format.class*org.apache.hadoop.mapred.SequenceFileOutputFormat
*https.keystore.info.rsrc*sslinfo.xml *
org.apache.mahout.clustering.kmeans.path*
/user/mdesnoyer/gamespot/boostcluster/boost0/init_clusters *
ipc.client.tcpnodelay*false *mapred.task.profile.reduces*0-2 *
mapred.output.compression.codec*org.apache.hadoop.io.compress.DefaultCodec
*io.map.index.skip*0 *mapred.working.dir*hdfs://
c17-dw-dev50-hdfs-dn-n1.corn.com/user/mdesnoyer *ipc.server.tcpnodelay*
false *mapred.reducer.class*
org.apache.mahout.clustering.kmeans.KMeansReducer *
dfs.default.chunk.view.size*32768 *hadoop.logfile.size*10000000 *
dfs.datanode.du.pct*0.98f *mapred.reduce.tasks.speculative.execution*true
*fs.inmemory.size.mb*75 *dfs.datanode.du.reserved*17179869184 *
fs.checkpoint.period*3600 *mapred.combiner.class*
org.apache.mahout.clustering.kmeans.KMeansCombiner *dfs.web.ugi*
webuser,webgroup *mapred.jobtracker.completeuserjobs.maximum*100 *
dfs.df.interval*60000 *dfs.data.dir*
/var/opt/hadoop/${hadoop.ident}/dfs/dfs0/data *fs.s3.maxRetries*4 *
dfs.datanode.dns.interface*default *mapred.mapoutput.value.class*
org.apache.mahout.clustering.kmeans.KMeansInfo *mapred.local.dir*
/var/opt/hadoop/${hadoop.ident}/mapred/mapred0/local *fs.hftp.impl*
org.apache.hadoop.dfs.HftpFileSystem *dfs.permissions.supergroup*httpds *
org.apache.mahout.clustering.kmeans.iteration*0 *mapred.mapper.class*
org.apache.mahout.clustering.kmeans.KMeansMapper *fs.trash.interval*0 *
fs.s3.sleepTimeSeconds*10 *dfs.replication.min*1 *
mapred.submit.replication*10 *fs.har.impl*
org.apache.hadoop.fs.HarFileSystem *mapred.map.output.compression.codec*
org.apache.hadoop.io.compress.DefaultCodec *
mapred.tasktracker.dns.interface*default *
dfs.namenode.decommission.interval*300 *dfs.http.address*0.0.0.0:50070 *
mapred.job.tracker*c17-dw-dev50-hdfs-dn-n3.corn.com:50031 *
dfs.heartbeat.interval*3 *io.seqfile.sorter.recordlimit*1000000 *
dfs.name.dir*/var/opt/hadoop/${hadoop.ident}/dfs/dfs0/name *
mapred.line.input.format.linespermap*1 *dfs.datanode.http.address*
0.0.0.0:50075 *mapred.local.dir.minspacekill*18253611008 *
dfs.replication.interval*3 *io.sort.record.percent*0.05 *
org.apache.mahout.clustering.kmeans.convergence*1.0E-4 *fs.kfs.impl*
org.apache.hadoop.fs.kfs.KosmosFileSystem *mapred.temp.dir*
${hadoop.tmp.dir}/mapred/temp *mapred.tasktracker.reduce.tasks.maximum*7 *
dfs.replication*3 *hadoop.logfile.count*10 *io.seqfile.compress.blocksize*
1000000 *fs.s3.block.size*67108864 *fs.hsftp.impl*
org.apache.hadoop.dfs.HsftpFileSystem *mapred.task.tracker.http.address*
0.0.0.0:50060 *mapred.reduce.parallel.copies*5 *io.seqfile.lazydecompress*
true *mapred.output.dir*hdfs://
c17-dw-dev50-hdfs-dn-n1.corn.com/user/mdesnoyer/... *io.sort.mb*100 *
ipc.client.connection.maxidletime*10000 *mapred.compress.map.output*false
*mapred.task.tracker.report.address*127.0.0.1:0 *ipc.client.kill.max*10 *
ipc.client.connect.max.retries*10 *fs.s3.impl*
org.apache.hadoop.fs.s3.S3FileSystem *mapred.input.dir*hdfs://
c17-dw-dev50-hdfs-dn-n1.corn.com/user/mdesnoyer/... *
mapred.job.tracker.http.address*0.0.0.0:50030 *io.file.buffer.size*4096 *
io.serializations*org.apache.hadoop.io.serializer.WritableSerialization *
dfs.datanode.handler.count*3 *mapred.reduce.copy.backoff*300 *
mapred.task.profile*false *dfs.replication.considerLoad*true *
jobclient.output.filter*FAILED *mapred.tasktracker.map.tasks.maximum*7 *
io.compression.codecs*
org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec
*fs.checkpoint.size*67108864
On Tue, Aug 4, 2009 at 10:37 AM, Grant Ingersoll <gs...@apache.org>wrote:
> Can you share your command line args, too?
>
>
>
> On Aug 4, 2009, at 10:23 AM, Mark Desnoyer wrote:
>
> Thanks Grant. I can't easily upgrade the Hadoop cluster that I'm using so
>> I
>> figured the mailing list was a good shot to see if anybody else had run
>> into
>> the same problem. If not (and that seems to be the case), then I really do
>> need to go bug the cluster maintainer to upgrade.
>>
>> Anyway, in case somebody has actually seen this, and for completeness,
>> here's the output from lsof.
>>
>> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
>> java 19625 mdesnoyer cwd DIR 8,1 4096 4138417
>> /home/mdesnoyer/src/graffiti0/summer09/hadoop
>> java 19625 mdesnoyer rtd DIR 8,1 4096 2 /
>> java 19625 mdesnoyer txt REG 8,1 47308 6645980
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>> java 19625 mdesnoyer mem REG 8,1 83552 2556065
>> /lib/libz.so.1.2.3.3
>> java 19625 mdesnoyer mem REG 8,1 1145980 6646075
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>> java 19625 mdesnoyer mem REG 8,1 102006 5243603
>> /opt/hadoop-0.18.3_dumbo/lib/native/Linux-i386-32/libhadoop.so
>> java 19625 mdesnoyer mem REG 8,1 75472 2573671
>> /lib/tls/i686/cmov/libresolv-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 21976 2573658
>> /lib/tls/i686/cmov/libnss_dns-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 36913 6646018
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnio.so
>> java 19625 mdesnoyer mem REG 8,1 33900 6646019
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libmanagement.so
>> java 19625 mdesnoyer mem REG 8,1 95801 6646017
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnet.so
>> java 19625 mdesnoyer mem REG 8,1 2145163 4138922
>> /home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
>> java 19625 mdesnoyer mem REG 8,1 50493 5243584
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
>> java 19625 mdesnoyer mem REG 8,1 105502 5243582
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
>> java 19625 mdesnoyer mem REG 8,1 351903 5243585
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
>> java 19625 mdesnoyer mem REG 8,1 112341 5243583
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
>> java 19625 mdesnoyer mem REG 8,1 15010 5243591
>> /opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
>> java 19625 mdesnoyer mem REG 8,1 8601 5243613
>> /opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
>> java 19625 mdesnoyer mem REG 8,1 15345 5243588
>> /opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
>> java 19625 mdesnoyer mem REG 8,1 97689 5243586
>> /opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
>> java 19625 mdesnoyer mem REG 8,1 65261 5243614
>> /opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
>> java 19625 mdesnoyer mem REG 8,1 391834 5243599
>> /opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
>> java 19625 mdesnoyer mem REG 8,1 9484 5243580
>> /opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
>> java 19625 mdesnoyer mem REG 8,1 121070 5243616
>> /opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
>> java 19625 mdesnoyer mem REG 8,1 665638 5243592
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
>> java 19625 mdesnoyer mem REG 8,1 288534 5243598
>> /opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
>> java 19625 mdesnoyer mem REG 8,1 180792 5243589
>> /opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
>> java 19625 mdesnoyer mem REG 8,1 26202 5243590
>> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
>> java 19625 mdesnoyer mem REG 8,1 38015 5243594
>> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
>> java 19625 mdesnoyer mem REG 8,1 279781 5243597
>> /opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
>> java 19625 mdesnoyer mem REG 8,1 46725 5243617
>> /opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
>> java 19625 mdesnoyer mem REG 8,1 258337 5243596
>> /opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
>> java 19625 mdesnoyer mem REG 8,1 1985528 3599303
>> /opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
>> java 19625 mdesnoyer mem REG 8,1 12588713 6645952
>> /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>> java 19625 mdesnoyer mem REG 8,1 256316 6276046
>> /usr/lib/locale/en_US.utf8/LC_CTYPE
>> java 19625 mdesnoyer mem REG 8,1 51669409 6646241
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>> java 19625 mdesnoyer mem REG 8,1 618950 6646646
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>> java 19625 mdesnoyer mem REG 8,1 76348 6646013
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libzip.so
>> java 19625 mdesnoyer mem REG 8,1 42504 2573660
>> /lib/tls/i686/cmov/libnss_files-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 38444 2573664
>> /lib/tls/i686/cmov/libnss_nis-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 30436 2573656
>> /lib/tls/i686/cmov/libnss_compat-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 26048 5652723
>> /usr/lib/gconv/gconv-modules.cache
>> java 19625 mdesnoyer mem REG 8,1 32768 3589971
>> /tmp/hsperfdata_mdesnoyer/19625
>> java 19625 mdesnoyer mem REG 8,1 87804 2573654
>> /lib/tls/i686/cmov/libnsl-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 38694 6645998
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/native_threads/libhpi.so
>> java 19625 mdesnoyer mem REG 8,1 189159 6646010
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libjava.so
>> java 19625 mdesnoyer mem REG 8,1 30624 2573673
>> /lib/tls/i686/cmov/librt-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 149328 2573651
>> /lib/tls/i686/cmov/libm-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 9180183 6646000
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
>> java 19625 mdesnoyer mem REG 8,1 1442180 2573643
>> /lib/tls/i686/cmov/libc-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 9676 2573649
>> /lib/tls/i686/cmov/libdl-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 38142 6646012
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/jli/libjli.so
>> java 19625 mdesnoyer mem REG 8,1 116405 2573669
>> /lib/tls/i686/cmov/libpthread-2.9.so
>> java 19625 mdesnoyer mem REG 8,1 7552 2555999
>> /lib/libnss_mdns4_minimal.so.2
>> java 19625 mdesnoyer mem REG 8,1 56701 6646009
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libverify.so
>> java 19625 mdesnoyer mem REG 8,1 117348 2555927 /lib/
>> ld-2.9.so
>> java 19625 mdesnoyer 0u CHR 136,0 2 /dev/pts/0
>> java 19625 mdesnoyer 1u CHR 136,0 2 /dev/pts/0
>> java 19625 mdesnoyer 2u CHR 136,0 2 /dev/pts/0
>> java 19625 mdesnoyer 3r REG 8,1 51669409 6646241
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>> java 19625 mdesnoyer 4r REG 8,1 12588713 6645952
>> /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>> java 19625 mdesnoyer 5r REG 8,1 1985528 3599303
>> /opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
>> java 19625 mdesnoyer 6r REG 8,1 258337 5243596
>> /opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
>> java 19625 mdesnoyer 7r REG 8,1 46725 5243617
>> /opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
>> java 19625 mdesnoyer 8r REG 8,1 279781 5243597
>> /opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
>> java 19625 mdesnoyer 9r REG 8,1 38015 5243594
>> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
>> java 19625 mdesnoyer 10r REG 8,1 26202 5243590
>> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
>> java 19625 mdesnoyer 11r REG 8,1 180792 5243589
>> /opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
>> java 19625 mdesnoyer 12r REG 8,1 288534 5243598
>> /opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
>> java 19625 mdesnoyer 13r REG 8,1 665638 5243592
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
>> java 19625 mdesnoyer 14r REG 8,1 121070 5243616
>> /opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
>> java 19625 mdesnoyer 15r REG 8,1 9484 5243580
>> /opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
>> java 19625 mdesnoyer 16r REG 8,1 391834 5243599
>> /opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
>> java 19625 mdesnoyer 17r REG 8,1 65261 5243614
>> /opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
>> java 19625 mdesnoyer 18r REG 8,1 97689 5243586
>> /opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
>> java 19625 mdesnoyer 19r REG 8,1 15345 5243588
>> /opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
>> java 19625 mdesnoyer 20r REG 8,1 8601 5243613
>> /opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
>> java 19625 mdesnoyer 21r REG 8,1 15010 5243591
>> /opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
>> java 19625 mdesnoyer 22r REG 8,1 112341 5243583
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
>> java 19625 mdesnoyer 23r REG 8,1 351903 5243585
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
>> java 19625 mdesnoyer 24r REG 8,1 105502 5243582
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
>> java 19625 mdesnoyer 25r REG 8,1 50493 5243584
>> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
>> java 19625 mdesnoyer 26r CHR 1,8 1817
>> /dev/random
>> java 19625 mdesnoyer 27r CHR 1,9 1825
>> /dev/urandom
>> java 19625 mdesnoyer 28r REG 8,1 2145163 4138922
>> /home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
>> java 19625 mdesnoyer 29r REG 8,1 618950 6646646
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>> java 19625 mdesnoyer 30r REG 8,1 1145980 6646075
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>> java 19625 mdesnoyer 31r FIFO 0,6 1344679 pipe
>> java 19625 mdesnoyer 32u sock 0,4 1344677 can't
>> identify protocol
>> java 19625 mdesnoyer 33w FIFO 0,6 1344679 pipe
>> java 19625 mdesnoyer 34u 0000 0,7 0 32 anon_inode
>> java 19625 mdesnoyer 35u unix 0xe280c700 1344680 socket
>> java 19625 mdesnoyer 36r IPv6 1348265 TCP
>> luna.corn.cnwk:42903->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 37w IPv6 1406254 TCP
>> luna.corn.cnwk:49999->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 38u IPv6 1407766 TCP
>> luna.corn.cnwk:51609->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 40u IPv6 1407979 TCP
>> luna.corn.cnwk:35437->c17-dw-dev50-hdfs-dn-n3.corn.com:50031(ESTABLISHED)
>> java 19625 mdesnoyer 41u IPv6 1347947 TCP
>> luna.corn.cnwk:39687->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 42r IPv6 1347948 TCP
>> luna.corn.cnwk:35215->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 43w IPv6 1347949 TCP
>> luna.corn.cnwk:35216->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 44u IPv6 1347950 TCP
>> luna.corn.cnwk:35217->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 48u IPv6 1347976 TCP
>> luna.corn.cnwk:35218->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 49u IPv6 1347977 TCP
>> luna.corn.cnwk:35219->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 50u IPv6 1347978 TCP
>> luna.corn.cnwk:42880->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 51u IPv6 1348006 TCP
>> luna.corn.cnwk:35221->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 52u IPv6 1348007 TCP
>> luna.corn.cnwk:42882->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 53u IPv6 1348034 TCP
>> luna.corn.cnwk:35223->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 54u IPv6 1348035 TCP
>> luna.corn.cnwk:35224->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 55u IPv6 1348036 TCP
>> luna.corn.cnwk:42885->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 56u IPv6 1348064 TCP
>> luna.corn.cnwk:35226->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 57u IPv6 1348065 TCP
>> luna.corn.cnwk:42887->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 58u IPv6 1348092 TCP
>> luna.corn.cnwk:39701->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 59u IPv6 1348093 TCP
>> luna.corn.cnwk:35229->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 60u IPv6 1348094 TCP
>> luna.corn.cnwk:42890->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 61u IPv6 1348122 TCP
>> luna.corn.cnwk:55555->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 62u IPv6 1348123 TCP
>> luna.corn.cnwk:35232->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 63u IPv6 1348150 TCP
>> luna.corn.cnwk:39706->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 64u IPv6 1348151 TCP
>> luna.corn.cnwk:35234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 65u IPv6 1348178 TCP
>> luna.corn.cnwk:42895->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 66u IPv6 1348179 TCP
>> luna.corn.cnwk:55560->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 67u IPv6 1348188 TCP
>> luna.corn.cnwk:42897->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 68u IPv6 1348208 TCP
>> luna.corn.cnwk:42898->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 69u IPv6 1348209 TCP
>> luna.corn.cnwk:55563->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 70u IPv6 1348236 TCP
>> luna.corn.cnwk:55564->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 71u IPv6 1348237 TCP
>> luna.corn.cnwk:55565->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 72u IPv6 1348264 TCP
>> luna.corn.cnwk:35242->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 73u IPv6 1348266 TCP
>> luna.corn.cnwk:35244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 74u IPv6 1348293 TCP
>> luna.corn.cnwk:39718->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 75u IPv6 1348294 TCP
>> luna.corn.cnwk:35246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 76u IPv6 1348321 TCP
>> luna.corn.cnwk:42907->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 77u IPv6 1348322 TCP
>> luna.corn.cnwk:42908->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 78u IPv6 1348325 TCP
>> luna.corn.cnwk:35249->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 79u IPv6 1348351 TCP
>> luna.corn.cnwk:42910->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 80u IPv6 1348352 TCP
>> luna.corn.cnwk:39724->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 81u IPv6 1348379 TCP
>> luna.corn.cnwk:39725->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 82u IPv6 1348380 TCP
>> luna.corn.cnwk:55577->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 83u IPv6 1348407 TCP
>> luna.corn.cnwk:39727->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 84u IPv6 1348408 TCP
>> luna.corn.cnwk:39728->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 85u IPv6 1348409 TCP
>> luna.corn.cnwk:35256->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 86u IPv6 1348410 TCP
>> luna.corn.cnwk:34784->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 88r IPv6 1406224 TCP
>> luna.corn.cnwk:56589->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 89u IPv6 1406225 TCP
>> luna.corn.cnwk:45601->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 90r IPv6 1406226 TCP
>> luna.corn.cnwk:45602->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 91w IPv6 1406227 TCP
>> luna.corn.cnwk:45603->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 92u IPv6 1406228 TCP
>> luna.corn.cnwk:49973->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 94w IPv6 1407722 TCP
>> luna.corn.cnwk:48650->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 95u IPv6 1407734 TCP
>> luna.corn.cnwk:48652->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 96u IPv6 1406229 TCP
>> luna.corn.cnwk:54989->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 97u IPv6 1406230 TCP
>> luna.corn.cnwk:56595->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 98u IPv6 1406231 TCP
>> luna.corn.cnwk:49976->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 99u IPv6 1406232 TCP
>> luna.corn.cnwk:49977->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 100u IPv6 1406233 TCP
>> luna.corn.cnwk:45609->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 101u IPv6 1406234 TCP
>> luna.corn.cnwk:56599->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 102u IPv6 1406235 TCP
>> luna.corn.cnwk:45611->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 103u IPv6 1406236 TCP
>> luna.corn.cnwk:54996->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 104u IPv6 1406237 TCP
>> luna.corn.cnwk:45613->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 105u IPv6 1406238 TCP
>> luna.corn.cnwk:56603->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 106u IPv6 1406239 TCP
>> luna.corn.cnwk:56604->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 107u IPv6 1406240 TCP
>> luna.corn.cnwk:45616->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 108u IPv6 1406241 TCP
>> luna.corn.cnwk:45617->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 109u IPv6 1406242 TCP
>> luna.corn.cnwk:49987->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 110u IPv6 1406243 TCP
>> luna.corn.cnwk:49988->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 111u IPv6 1406244 TCP
>> luna.corn.cnwk:56609->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 112u IPv6 1406245 TCP
>> luna.corn.cnwk:45621->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 113u IPv6 1406246 TCP
>> luna.corn.cnwk:45622->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 114u IPv6 1406247 TCP
>> luna.corn.cnwk:55007->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 115u IPv6 1406248 TCP
>> luna.corn.cnwk:45624->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 116u IPv6 1406249 TCP
>> luna.corn.cnwk:55009->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 117u IPv6 1406250 TCP
>> luna.corn.cnwk:56615->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 118u IPv6 1406251 TCP
>> luna.corn.cnwk:55011->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 119u IPv6 1406252 TCP
>> luna.corn.cnwk:56617->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 120u IPv6 1406253 TCP
>> luna.corn.cnwk:55013->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 121u IPv6 1406255 TCP
>> luna.corn.cnwk:45631->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 122u IPv6 1406256 TCP
>> luna.corn.cnwk:56621->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 123u IPv6 1406257 TCP
>> luna.corn.cnwk:45633->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 124u IPv6 1406258 TCP
>> luna.corn.cnwk:50003->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 125u IPv6 1406259 TCP
>> luna.corn.cnwk:45635->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 126u IPv6 1406260 TCP
>> luna.corn.cnwk:56625->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 127u IPv6 1406261 TCP
>> luna.corn.cnwk:45637->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 128u IPv6 1406262 TCP
>> luna.corn.cnwk:55022->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 129u IPv6 1406263 TCP
>> luna.corn.cnwk:45639->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 130u IPv6 1406264 TCP
>> luna.corn.cnwk:56629->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 131u IPv6 1406265 TCP
>> luna.corn.cnwk:50010->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 132u IPv6 1406266 TCP
>> luna.corn.cnwk:56631->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 133u IPv6 1406267 TCP
>> luna.corn.cnwk:50012->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 134u IPv6 1407738 TCP
>> luna.corn.cnwk:42171->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 135u IPv6 1407739 TCP
>> luna.corn.cnwk:42172->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 136u IPv6 1407740 TCP
>> luna.corn.cnwk:48658->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 137u IPv6 1407741 TCP
>> luna.corn.cnwk:42174->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 138u IPv6 1407742 TCP
>> luna.corn.cnwk:51586->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 139u IPv6 1407743 TCP
>> luna.corn.cnwk:48661->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 143u IPv6 1407744 TCP
>> luna.corn.cnwk:51588->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 144u IPv6 1407745 TCP
>> luna.corn.cnwk:42178->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 145u IPv6 1407746 TCP
>> luna.corn.cnwk:54228->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 146u IPv6 1407747 TCP
>> luna.corn.cnwk:51591->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 147u IPv6 1407748 TCP
>> luna.corn.cnwk:48666->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 148u IPv6 1407749 TCP
>> luna.corn.cnwk:48667->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 149u IPv6 1407750 TCP
>> luna.corn.cnwk:42183->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 150u IPv6 1407751 TCP
>> luna.corn.cnwk:51595->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 151u IPv6 1407752 TCP
>> luna.corn.cnwk:54234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 152u IPv6 1407753 TCP
>> luna.corn.cnwk:48671->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 153u IPv6 1407754 TCP
>> luna.corn.cnwk:48672->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 154u IPv6 1407755 TCP
>> luna.corn.cnwk:42188->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 155u IPv6 1407756 TCP
>> luna.corn.cnwk:54238->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 156u IPv6 1407757 TCP
>> luna.corn.cnwk:42190->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 157u IPv6 1407758 TCP
>> luna.corn.cnwk:51602->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 158u IPv6 1407759 TCP
>> luna.corn.cnwk:48677->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 159u IPv6 1407760 TCP
>> luna.corn.cnwk:42193->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 160u IPv6 1407761 TCP
>> luna.corn.cnwk:54243->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 161u IPv6 1407762 TCP
>> luna.corn.cnwk:54244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 162u IPv6 1407763 TCP
>> luna.corn.cnwk:42196->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 163u IPv6 1407764 TCP
>> luna.corn.cnwk:54246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 164u IPv6 1407767 TCP
>> luna.corn.cnwk:51610->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 165u IPv6 1407768 TCP
>> luna.corn.cnwk:42200->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 166u IPv6 1407769 TCP
>> luna.corn.cnwk:51612->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 167u IPv6 1407770 TCP
>> luna.corn.cnwk:51613->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 168u IPv6 1407771 TCP
>> luna.corn.cnwk:48688->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 169u IPv6 1407772 TCP
>> luna.corn.cnwk:42204->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 170u IPv6 1407773 TCP
>> luna.corn.cnwk:42205->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 171u IPv6 1407774 TCP
>> luna.corn.cnwk:51617->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 172u IPv6 1407775 TCP
>> luna.corn.cnwk:48692->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 173u IPv6 1407776 TCP
>> luna.corn.cnwk:42208->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 174u IPv6 1407777 TCP
>> luna.corn.cnwk:54258->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 175u IPv6 1407778 TCP
>> luna.corn.cnwk:51621->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 176u IPv6 1407779 TCP
>> luna.corn.cnwk:54260->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 177u IPv6 1407780 TCP
>> luna.corn.cnwk:48697->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
>> java 19625 mdesnoyer 178r FIFO 0,6 1407966 pipe
>> java 19625 mdesnoyer 179w FIFO 0,6 1407966 pipe
>> java 19625 mdesnoyer 180u 0000 0,7 0 32 anon_inode
>>
>> On Mon, Aug 3, 2009 at 6:10 PM, Grant Ingersoll <gs...@apache.org>
>> wrote:
>>
>> Attachments usually don't come through on ASF mailing lists.
>>>
>>> Also, Mahout trunk requires Hadoop 0.20.0
>>>
>>>
>>>
>>> On Aug 3, 2009, at 6:05 PM, Mark Desnoyer wrote:
>>>
>>> Hi,
>>>
>>>>
>>>> Has anybody had a problem running KMeans leaking TCP connections, which
>>>> eventually causes the OS to issue a "Too Many Files Open" error? In
>>>> other
>>>> words, it looks like a file descriptor leak.
>>>>
>>>> I'm running a four machine cluster of Hadoop 18.3, with the current
>>>> version of Mahout from the trunk. Each k-means iteration is causing an
>>>> extra
>>>> 40 or so TCP connections to be created that get orphaned in CLOSE_WAIT.
>>>> I've
>>>> attached the output of lsof from the middle of a run.
>>>>
>>>> Any help would be appreciated!
>>>>
>>>> Cheers,
>>>> Mark
>>>>
>>>>
>>>
>>>
>>>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
> Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
Re: KMeans Leaked TCP Connections
Posted by Grant Ingersoll <gs...@apache.org>.
Can you share your command line args, too?
On Aug 4, 2009, at 10:23 AM, Mark Desnoyer wrote:
> Thanks Grant. I can't easily upgrade the Hadoop cluster that I'm
> using so I
> figured the mailing list was a good shot to see if anybody else had
> run into
> the same problem. If not (and that seems to be the case), then I
> really do
> need to go bug the cluster maintainer to upgrade.
>
> Anyway, in case somebody has actually seen this, and for completeness,
> here's the output from lsof.
>
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> java 19625 mdesnoyer cwd DIR 8,1 4096 4138417
> /home/mdesnoyer/src/graffiti0/summer09/hadoop
> java 19625 mdesnoyer rtd DIR 8,1 4096 2 /
> java 19625 mdesnoyer txt REG 8,1 47308 6645980
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
> java 19625 mdesnoyer mem REG 8,1 83552 2556065
> /lib/libz.so.1.2.3.3
> java 19625 mdesnoyer mem REG 8,1 1145980 6646075
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
> java 19625 mdesnoyer mem REG 8,1 102006 5243603
> /opt/hadoop-0.18.3_dumbo/lib/native/Linux-i386-32/libhadoop.so
> java 19625 mdesnoyer mem REG 8,1 75472 2573671
> /lib/tls/i686/cmov/libresolv-2.9.so
> java 19625 mdesnoyer mem REG 8,1 21976 2573658
> /lib/tls/i686/cmov/libnss_dns-2.9.so
> java 19625 mdesnoyer mem REG 8,1 36913 6646018
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnio.so
> java 19625 mdesnoyer mem REG 8,1 33900 6646019
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libmanagement.so
> java 19625 mdesnoyer mem REG 8,1 95801 6646017
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnet.so
> java 19625 mdesnoyer mem REG 8,1 2145163 4138922
> /home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
> java 19625 mdesnoyer mem REG 8,1 50493 5243584
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
> java 19625 mdesnoyer mem REG 8,1 105502 5243582
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
> java 19625 mdesnoyer mem REG 8,1 351903 5243585
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
> java 19625 mdesnoyer mem REG 8,1 112341 5243583
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
> java 19625 mdesnoyer mem REG 8,1 15010 5243591
> /opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
> java 19625 mdesnoyer mem REG 8,1 8601 5243613
> /opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
> java 19625 mdesnoyer mem REG 8,1 15345 5243588
> /opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
> java 19625 mdesnoyer mem REG 8,1 97689 5243586
> /opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
> java 19625 mdesnoyer mem REG 8,1 65261 5243614
> /opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
> java 19625 mdesnoyer mem REG 8,1 391834 5243599
> /opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
> java 19625 mdesnoyer mem REG 8,1 9484 5243580
> /opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
> java 19625 mdesnoyer mem REG 8,1 121070 5243616
> /opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
> java 19625 mdesnoyer mem REG 8,1 665638 5243592
> /opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
> java 19625 mdesnoyer mem REG 8,1 288534 5243598
> /opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
> java 19625 mdesnoyer mem REG 8,1 180792 5243589
> /opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
> java 19625 mdesnoyer mem REG 8,1 26202 5243590
> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
> java 19625 mdesnoyer mem REG 8,1 38015 5243594
> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
> java 19625 mdesnoyer mem REG 8,1 279781 5243597
> /opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
> java 19625 mdesnoyer mem REG 8,1 46725 5243617
> /opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
> java 19625 mdesnoyer mem REG 8,1 258337 5243596
> /opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
> java 19625 mdesnoyer mem REG 8,1 1985528 3599303
> /opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
> java 19625 mdesnoyer mem REG 8,1 12588713 6645952
> /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
> java 19625 mdesnoyer mem REG 8,1 256316 6276046
> /usr/lib/locale/en_US.utf8/LC_CTYPE
> java 19625 mdesnoyer mem REG 8,1 51669409 6646241
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
> java 19625 mdesnoyer mem REG 8,1 618950 6646646
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
> java 19625 mdesnoyer mem REG 8,1 76348 6646013
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libzip.so
> java 19625 mdesnoyer mem REG 8,1 42504 2573660
> /lib/tls/i686/cmov/libnss_files-2.9.so
> java 19625 mdesnoyer mem REG 8,1 38444 2573664
> /lib/tls/i686/cmov/libnss_nis-2.9.so
> java 19625 mdesnoyer mem REG 8,1 30436 2573656
> /lib/tls/i686/cmov/libnss_compat-2.9.so
> java 19625 mdesnoyer mem REG 8,1 26048 5652723
> /usr/lib/gconv/gconv-modules.cache
> java 19625 mdesnoyer mem REG 8,1 32768 3589971
> /tmp/hsperfdata_mdesnoyer/19625
> java 19625 mdesnoyer mem REG 8,1 87804 2573654
> /lib/tls/i686/cmov/libnsl-2.9.so
> java 19625 mdesnoyer mem REG 8,1 38694 6645998
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/native_threads/libhpi.so
> java 19625 mdesnoyer mem REG 8,1 189159 6646010
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libjava.so
> java 19625 mdesnoyer mem REG 8,1 30624 2573673
> /lib/tls/i686/cmov/librt-2.9.so
> java 19625 mdesnoyer mem REG 8,1 149328 2573651
> /lib/tls/i686/cmov/libm-2.9.so
> java 19625 mdesnoyer mem REG 8,1 9180183 6646000
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
> java 19625 mdesnoyer mem REG 8,1 1442180 2573643
> /lib/tls/i686/cmov/libc-2.9.so
> java 19625 mdesnoyer mem REG 8,1 9676 2573649
> /lib/tls/i686/cmov/libdl-2.9.so
> java 19625 mdesnoyer mem REG 8,1 38142 6646012
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/jli/libjli.so
> java 19625 mdesnoyer mem REG 8,1 116405 2573669
> /lib/tls/i686/cmov/libpthread-2.9.so
> java 19625 mdesnoyer mem REG 8,1 7552 2555999
> /lib/libnss_mdns4_minimal.so.2
> java 19625 mdesnoyer mem REG 8,1 56701 6646009
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libverify.so
> java 19625 mdesnoyer mem REG 8,1 117348 2555927 /lib/
> ld-2.9.so
> java 19625 mdesnoyer 0u CHR 136,0 2 /dev/
> pts/0
> java 19625 mdesnoyer 1u CHR 136,0 2 /dev/
> pts/0
> java 19625 mdesnoyer 2u CHR 136,0 2 /dev/
> pts/0
> java 19625 mdesnoyer 3r REG 8,1 51669409 6646241
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
> java 19625 mdesnoyer 4r REG 8,1 12588713 6645952
> /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
> java 19625 mdesnoyer 5r REG 8,1 1985528 3599303
> /opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
> java 19625 mdesnoyer 6r REG 8,1 258337 5243596
> /opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
> java 19625 mdesnoyer 7r REG 8,1 46725 5243617
> /opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
> java 19625 mdesnoyer 8r REG 8,1 279781 5243597
> /opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
> java 19625 mdesnoyer 9r REG 8,1 38015 5243594
> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
> java 19625 mdesnoyer 10r REG 8,1 26202 5243590
> /opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
> java 19625 mdesnoyer 11r REG 8,1 180792 5243589
> /opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
> java 19625 mdesnoyer 12r REG 8,1 288534 5243598
> /opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
> java 19625 mdesnoyer 13r REG 8,1 665638 5243592
> /opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
> java 19625 mdesnoyer 14r REG 8,1 121070 5243616
> /opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
> java 19625 mdesnoyer 15r REG 8,1 9484 5243580
> /opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
> java 19625 mdesnoyer 16r REG 8,1 391834 5243599
> /opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
> java 19625 mdesnoyer 17r REG 8,1 65261 5243614
> /opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
> java 19625 mdesnoyer 18r REG 8,1 97689 5243586
> /opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
> java 19625 mdesnoyer 19r REG 8,1 15345 5243588
> /opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
> java 19625 mdesnoyer 20r REG 8,1 8601 5243613
> /opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
> java 19625 mdesnoyer 21r REG 8,1 15010 5243591
> /opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
> java 19625 mdesnoyer 22r REG 8,1 112341 5243583
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
> java 19625 mdesnoyer 23r REG 8,1 351903 5243585
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
> java 19625 mdesnoyer 24r REG 8,1 105502 5243582
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
> java 19625 mdesnoyer 25r REG 8,1 50493 5243584
> /opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
> java 19625 mdesnoyer 26r CHR 1,8 1817 /dev/
> random
> java 19625 mdesnoyer 27r CHR 1,9 1825 /dev/
> urandom
> java 19625 mdesnoyer 28r REG 8,1 2145163 4138922
> /home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
> java 19625 mdesnoyer 29r REG 8,1 618950 6646646
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
> java 19625 mdesnoyer 30r REG 8,1 1145980 6646075
> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
> java 19625 mdesnoyer 31r FIFO 0,6 1344679 pipe
> java 19625 mdesnoyer 32u sock 0,4 1344677 can't
> identify protocol
> java 19625 mdesnoyer 33w FIFO 0,6 1344679 pipe
> java 19625 mdesnoyer 34u 0000 0,7 0 32
> anon_inode
> java 19625 mdesnoyer 35u unix 0xe280c700 1344680 socket
> java 19625 mdesnoyer 36r IPv6 1348265 TCP
> luna.corn.cnwk:42903->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 37w IPv6 1406254 TCP
> luna.corn.cnwk:49999->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 38u IPv6 1407766 TCP
> luna.corn.cnwk:51609->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 40u IPv6 1407979 TCP
> luna.corn.cnwk:35437->c17-dw-dev50-hdfs-dn-n3.corn.com:50031
> (ESTABLISHED)
> java 19625 mdesnoyer 41u IPv6 1347947 TCP
> luna.corn.cnwk:39687->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 42r IPv6 1347948 TCP
> luna.corn.cnwk:35215->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 43w IPv6 1347949 TCP
> luna.corn.cnwk:35216->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 44u IPv6 1347950 TCP
> luna.corn.cnwk:35217->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 48u IPv6 1347976 TCP
> luna.corn.cnwk:35218->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 49u IPv6 1347977 TCP
> luna.corn.cnwk:35219->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 50u IPv6 1347978 TCP
> luna.corn.cnwk:42880->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 51u IPv6 1348006 TCP
> luna.corn.cnwk:35221->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 52u IPv6 1348007 TCP
> luna.corn.cnwk:42882->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 53u IPv6 1348034 TCP
> luna.corn.cnwk:35223->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 54u IPv6 1348035 TCP
> luna.corn.cnwk:35224->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 55u IPv6 1348036 TCP
> luna.corn.cnwk:42885->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 56u IPv6 1348064 TCP
> luna.corn.cnwk:35226->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 57u IPv6 1348065 TCP
> luna.corn.cnwk:42887->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 58u IPv6 1348092 TCP
> luna.corn.cnwk:39701->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 59u IPv6 1348093 TCP
> luna.corn.cnwk:35229->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 60u IPv6 1348094 TCP
> luna.corn.cnwk:42890->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 61u IPv6 1348122 TCP
> luna.corn.cnwk:55555->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 62u IPv6 1348123 TCP
> luna.corn.cnwk:35232->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 63u IPv6 1348150 TCP
> luna.corn.cnwk:39706->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 64u IPv6 1348151 TCP
> luna.corn.cnwk:35234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 65u IPv6 1348178 TCP
> luna.corn.cnwk:42895->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 66u IPv6 1348179 TCP
> luna.corn.cnwk:55560->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 67u IPv6 1348188 TCP
> luna.corn.cnwk:42897->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 68u IPv6 1348208 TCP
> luna.corn.cnwk:42898->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 69u IPv6 1348209 TCP
> luna.corn.cnwk:55563->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 70u IPv6 1348236 TCP
> luna.corn.cnwk:55564->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 71u IPv6 1348237 TCP
> luna.corn.cnwk:55565->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 72u IPv6 1348264 TCP
> luna.corn.cnwk:35242->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 73u IPv6 1348266 TCP
> luna.corn.cnwk:35244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 74u IPv6 1348293 TCP
> luna.corn.cnwk:39718->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 75u IPv6 1348294 TCP
> luna.corn.cnwk:35246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 76u IPv6 1348321 TCP
> luna.corn.cnwk:42907->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 77u IPv6 1348322 TCP
> luna.corn.cnwk:42908->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 78u IPv6 1348325 TCP
> luna.corn.cnwk:35249->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 79u IPv6 1348351 TCP
> luna.corn.cnwk:42910->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 80u IPv6 1348352 TCP
> luna.corn.cnwk:39724->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 81u IPv6 1348379 TCP
> luna.corn.cnwk:39725->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 82u IPv6 1348380 TCP
> luna.corn.cnwk:55577->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 83u IPv6 1348407 TCP
> luna.corn.cnwk:39727->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 84u IPv6 1348408 TCP
> luna.corn.cnwk:39728->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 85u IPv6 1348409 TCP
> luna.corn.cnwk:35256->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 86u IPv6 1348410 TCP
> luna.corn.cnwk:34784->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 88r IPv6 1406224 TCP
> luna.corn.cnwk:56589->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 89u IPv6 1406225 TCP
> luna.corn.cnwk:45601->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 90r IPv6 1406226 TCP
> luna.corn.cnwk:45602->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 91w IPv6 1406227 TCP
> luna.corn.cnwk:45603->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 92u IPv6 1406228 TCP
> luna.corn.cnwk:49973->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 94w IPv6 1407722 TCP
> luna.corn.cnwk:48650->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 95u IPv6 1407734 TCP
> luna.corn.cnwk:48652->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 96u IPv6 1406229 TCP
> luna.corn.cnwk:54989->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 97u IPv6 1406230 TCP
> luna.corn.cnwk:56595->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 98u IPv6 1406231 TCP
> luna.corn.cnwk:49976->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 99u IPv6 1406232 TCP
> luna.corn.cnwk:49977->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 100u IPv6 1406233 TCP
> luna.corn.cnwk:45609->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 101u IPv6 1406234 TCP
> luna.corn.cnwk:56599->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 102u IPv6 1406235 TCP
> luna.corn.cnwk:45611->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 103u IPv6 1406236 TCP
> luna.corn.cnwk:54996->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 104u IPv6 1406237 TCP
> luna.corn.cnwk:45613->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 105u IPv6 1406238 TCP
> luna.corn.cnwk:56603->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 106u IPv6 1406239 TCP
> luna.corn.cnwk:56604->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 107u IPv6 1406240 TCP
> luna.corn.cnwk:45616->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 108u IPv6 1406241 TCP
> luna.corn.cnwk:45617->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 109u IPv6 1406242 TCP
> luna.corn.cnwk:49987->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 110u IPv6 1406243 TCP
> luna.corn.cnwk:49988->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 111u IPv6 1406244 TCP
> luna.corn.cnwk:56609->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 112u IPv6 1406245 TCP
> luna.corn.cnwk:45621->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 113u IPv6 1406246 TCP
> luna.corn.cnwk:45622->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 114u IPv6 1406247 TCP
> luna.corn.cnwk:55007->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 115u IPv6 1406248 TCP
> luna.corn.cnwk:45624->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 116u IPv6 1406249 TCP
> luna.corn.cnwk:55009->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 117u IPv6 1406250 TCP
> luna.corn.cnwk:56615->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 118u IPv6 1406251 TCP
> luna.corn.cnwk:55011->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 119u IPv6 1406252 TCP
> luna.corn.cnwk:56617->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 120u IPv6 1406253 TCP
> luna.corn.cnwk:55013->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 121u IPv6 1406255 TCP
> luna.corn.cnwk:45631->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 122u IPv6 1406256 TCP
> luna.corn.cnwk:56621->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 123u IPv6 1406257 TCP
> luna.corn.cnwk:45633->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 124u IPv6 1406258 TCP
> luna.corn.cnwk:50003->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 125u IPv6 1406259 TCP
> luna.corn.cnwk:45635->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 126u IPv6 1406260 TCP
> luna.corn.cnwk:56625->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 127u IPv6 1406261 TCP
> luna.corn.cnwk:45637->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 128u IPv6 1406262 TCP
> luna.corn.cnwk:55022->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 129u IPv6 1406263 TCP
> luna.corn.cnwk:45639->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 130u IPv6 1406264 TCP
> luna.corn.cnwk:56629->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 131u IPv6 1406265 TCP
> luna.corn.cnwk:50010->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 132u IPv6 1406266 TCP
> luna.corn.cnwk:56631->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 133u IPv6 1406267 TCP
> luna.corn.cnwk:50012->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 134u IPv6 1407738 TCP
> luna.corn.cnwk:42171->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 135u IPv6 1407739 TCP
> luna.corn.cnwk:42172->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 136u IPv6 1407740 TCP
> luna.corn.cnwk:48658->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 137u IPv6 1407741 TCP
> luna.corn.cnwk:42174->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 138u IPv6 1407742 TCP
> luna.corn.cnwk:51586->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 139u IPv6 1407743 TCP
> luna.corn.cnwk:48661->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 143u IPv6 1407744 TCP
> luna.corn.cnwk:51588->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 144u IPv6 1407745 TCP
> luna.corn.cnwk:42178->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 145u IPv6 1407746 TCP
> luna.corn.cnwk:54228->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 146u IPv6 1407747 TCP
> luna.corn.cnwk:51591->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 147u IPv6 1407748 TCP
> luna.corn.cnwk:48666->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 148u IPv6 1407749 TCP
> luna.corn.cnwk:48667->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 149u IPv6 1407750 TCP
> luna.corn.cnwk:42183->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 150u IPv6 1407751 TCP
> luna.corn.cnwk:51595->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 151u IPv6 1407752 TCP
> luna.corn.cnwk:54234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 152u IPv6 1407753 TCP
> luna.corn.cnwk:48671->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 153u IPv6 1407754 TCP
> luna.corn.cnwk:48672->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 154u IPv6 1407755 TCP
> luna.corn.cnwk:42188->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 155u IPv6 1407756 TCP
> luna.corn.cnwk:54238->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 156u IPv6 1407757 TCP
> luna.corn.cnwk:42190->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 157u IPv6 1407758 TCP
> luna.corn.cnwk:51602->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 158u IPv6 1407759 TCP
> luna.corn.cnwk:48677->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 159u IPv6 1407760 TCP
> luna.corn.cnwk:42193->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 160u IPv6 1407761 TCP
> luna.corn.cnwk:54243->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 161u IPv6 1407762 TCP
> luna.corn.cnwk:54244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 162u IPv6 1407763 TCP
> luna.corn.cnwk:42196->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 163u IPv6 1407764 TCP
> luna.corn.cnwk:54246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 164u IPv6 1407767 TCP
> luna.corn.cnwk:51610->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 165u IPv6 1407768 TCP
> luna.corn.cnwk:42200->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 166u IPv6 1407769 TCP
> luna.corn.cnwk:51612->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 167u IPv6 1407770 TCP
> luna.corn.cnwk:51613->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 168u IPv6 1407771 TCP
> luna.corn.cnwk:48688->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 169u IPv6 1407772 TCP
> luna.corn.cnwk:42204->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 170u IPv6 1407773 TCP
> luna.corn.cnwk:42205->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 171u IPv6 1407774 TCP
> luna.corn.cnwk:51617->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 172u IPv6 1407775 TCP
> luna.corn.cnwk:48692->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 173u IPv6 1407776 TCP
> luna.corn.cnwk:42208->c17-dw-dev50-hdfs-dn-n1.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 174u IPv6 1407777 TCP
> luna.corn.cnwk:54258->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 175u IPv6 1407778 TCP
> luna.corn.cnwk:51621->c17-dw-dev50-hdfs-dn-n4.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 176u IPv6 1407779 TCP
> luna.corn.cnwk:54260->c17-dw-dev50-hdfs-dn-n2.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 177u IPv6 1407780 TCP
> luna.corn.cnwk:48697->c17-dw-dev50-hdfs-dn-n3.corn.com:50010
> (CLOSE_WAIT)
> java 19625 mdesnoyer 178r FIFO 0,6 1407966 pipe
> java 19625 mdesnoyer 179w FIFO 0,6 1407966 pipe
> java 19625 mdesnoyer 180u 0000 0,7 0 32
> anon_inode
>
> On Mon, Aug 3, 2009 at 6:10 PM, Grant Ingersoll
> <gs...@apache.org> wrote:
>
>> Attachments usually don't come through on ASF mailing lists.
>>
>> Also, Mahout trunk requires Hadoop 0.20.0
>>
>>
>>
>> On Aug 3, 2009, at 6:05 PM, Mark Desnoyer wrote:
>>
>> Hi,
>>>
>>> Has anybody had a problem running KMeans leaking TCP connections,
>>> which
>>> eventually causes the OS to issue a "Too Many Files Open" error?
>>> In other
>>> words, it looks like a file descriptor leak.
>>>
>>> I'm running a four machine cluster of Hadoop 18.3, with the current
>>> version of Mahout from the trunk. Each k-means iteration is
>>> causing an extra
>>> 40 or so TCP connections to be created that get orphaned in
>>> CLOSE_WAIT. I've
>>> attached the output of lsof from the middle of a run.
>>>
>>> Any help would be appreciated!
>>>
>>> Cheers,
>>> Mark
>>>
>>
>>
>>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: KMeans Leaked TCP Connections
Posted by Mark Desnoyer <md...@gmail.com>.
Thanks Grant. I can't easily upgrade the Hadoop cluster that I'm using so I
figured the mailing list was a good shot to see if anybody else had run into
the same problem. If not (and that seems to be the case), then I really do
need to go bug the cluster maintainer to upgrade.
Anyway, in case somebody has actually seen this, and for completeness,
here's the output from lsof.
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
java 19625 mdesnoyer cwd DIR 8,1 4096 4138417
/home/mdesnoyer/src/graffiti0/summer09/hadoop
java 19625 mdesnoyer rtd DIR 8,1 4096 2 /
java 19625 mdesnoyer txt REG 8,1 47308 6645980
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
java 19625 mdesnoyer mem REG 8,1 83552 2556065
/lib/libz.so.1.2.3.3
java 19625 mdesnoyer mem REG 8,1 1145980 6646075
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
java 19625 mdesnoyer mem REG 8,1 102006 5243603
/opt/hadoop-0.18.3_dumbo/lib/native/Linux-i386-32/libhadoop.so
java 19625 mdesnoyer mem REG 8,1 75472 2573671
/lib/tls/i686/cmov/libresolv-2.9.so
java 19625 mdesnoyer mem REG 8,1 21976 2573658
/lib/tls/i686/cmov/libnss_dns-2.9.so
java 19625 mdesnoyer mem REG 8,1 36913 6646018
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnio.so
java 19625 mdesnoyer mem REG 8,1 33900 6646019
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libmanagement.so
java 19625 mdesnoyer mem REG 8,1 95801 6646017
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libnet.so
java 19625 mdesnoyer mem REG 8,1 2145163 4138922
/home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
java 19625 mdesnoyer mem REG 8,1 50493 5243584
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
java 19625 mdesnoyer mem REG 8,1 105502 5243582
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
java 19625 mdesnoyer mem REG 8,1 351903 5243585
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
java 19625 mdesnoyer mem REG 8,1 112341 5243583
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
java 19625 mdesnoyer mem REG 8,1 15010 5243591
/opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
java 19625 mdesnoyer mem REG 8,1 8601 5243613
/opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
java 19625 mdesnoyer mem REG 8,1 15345 5243588
/opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
java 19625 mdesnoyer mem REG 8,1 97689 5243586
/opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
java 19625 mdesnoyer mem REG 8,1 65261 5243614
/opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
java 19625 mdesnoyer mem REG 8,1 391834 5243599
/opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
java 19625 mdesnoyer mem REG 8,1 9484 5243580
/opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
java 19625 mdesnoyer mem REG 8,1 121070 5243616
/opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
java 19625 mdesnoyer mem REG 8,1 665638 5243592
/opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
java 19625 mdesnoyer mem REG 8,1 288534 5243598
/opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
java 19625 mdesnoyer mem REG 8,1 180792 5243589
/opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
java 19625 mdesnoyer mem REG 8,1 26202 5243590
/opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
java 19625 mdesnoyer mem REG 8,1 38015 5243594
/opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
java 19625 mdesnoyer mem REG 8,1 279781 5243597
/opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
java 19625 mdesnoyer mem REG 8,1 46725 5243617
/opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
java 19625 mdesnoyer mem REG 8,1 258337 5243596
/opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
java 19625 mdesnoyer mem REG 8,1 1985528 3599303
/opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
java 19625 mdesnoyer mem REG 8,1 12588713 6645952
/usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
java 19625 mdesnoyer mem REG 8,1 256316 6276046
/usr/lib/locale/en_US.utf8/LC_CTYPE
java 19625 mdesnoyer mem REG 8,1 51669409 6646241
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
java 19625 mdesnoyer mem REG 8,1 618950 6646646
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
java 19625 mdesnoyer mem REG 8,1 76348 6646013
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libzip.so
java 19625 mdesnoyer mem REG 8,1 42504 2573660
/lib/tls/i686/cmov/libnss_files-2.9.so
java 19625 mdesnoyer mem REG 8,1 38444 2573664
/lib/tls/i686/cmov/libnss_nis-2.9.so
java 19625 mdesnoyer mem REG 8,1 30436 2573656
/lib/tls/i686/cmov/libnss_compat-2.9.so
java 19625 mdesnoyer mem REG 8,1 26048 5652723
/usr/lib/gconv/gconv-modules.cache
java 19625 mdesnoyer mem REG 8,1 32768 3589971
/tmp/hsperfdata_mdesnoyer/19625
java 19625 mdesnoyer mem REG 8,1 87804 2573654
/lib/tls/i686/cmov/libnsl-2.9.so
java 19625 mdesnoyer mem REG 8,1 38694 6645998
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/native_threads/libhpi.so
java 19625 mdesnoyer mem REG 8,1 189159 6646010
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libjava.so
java 19625 mdesnoyer mem REG 8,1 30624 2573673
/lib/tls/i686/cmov/librt-2.9.so
java 19625 mdesnoyer mem REG 8,1 149328 2573651
/lib/tls/i686/cmov/libm-2.9.so
java 19625 mdesnoyer mem REG 8,1 9180183 6646000
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
java 19625 mdesnoyer mem REG 8,1 1442180 2573643
/lib/tls/i686/cmov/libc-2.9.so
java 19625 mdesnoyer mem REG 8,1 9676 2573649
/lib/tls/i686/cmov/libdl-2.9.so
java 19625 mdesnoyer mem REG 8,1 38142 6646012
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/jli/libjli.so
java 19625 mdesnoyer mem REG 8,1 116405 2573669
/lib/tls/i686/cmov/libpthread-2.9.so
java 19625 mdesnoyer mem REG 8,1 7552 2555999
/lib/libnss_mdns4_minimal.so.2
java 19625 mdesnoyer mem REG 8,1 56701 6646009
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/libverify.so
java 19625 mdesnoyer mem REG 8,1 117348 2555927 /lib/
ld-2.9.so
java 19625 mdesnoyer 0u CHR 136,0 2 /dev/pts/0
java 19625 mdesnoyer 1u CHR 136,0 2 /dev/pts/0
java 19625 mdesnoyer 2u CHR 136,0 2 /dev/pts/0
java 19625 mdesnoyer 3r REG 8,1 51669409 6646241
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
java 19625 mdesnoyer 4r REG 8,1 12588713 6645952
/usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
java 19625 mdesnoyer 5r REG 8,1 1985528 3599303
/opt/hadoop-0.18.3_dumbo/hadoop-0.18.3_dumbo-core.jar
java 19625 mdesnoyer 6r REG 8,1 258337 5243596
/opt/hadoop-0.18.3_dumbo/lib/commons-cli-2.0-SNAPSHOT.jar
java 19625 mdesnoyer 7r REG 8,1 46725 5243617
/opt/hadoop-0.18.3_dumbo/lib/commons-codec-1.3.jar
java 19625 mdesnoyer 8r REG 8,1 279781 5243597
/opt/hadoop-0.18.3_dumbo/lib/commons-httpclient-3.0.1.jar
java 19625 mdesnoyer 9r REG 8,1 38015 5243594
/opt/hadoop-0.18.3_dumbo/lib/commons-logging-1.0.4.jar
java 19625 mdesnoyer 10r REG 8,1 26202 5243590
/opt/hadoop-0.18.3_dumbo/lib/commons-logging-api-1.0.4.jar
java 19625 mdesnoyer 11r REG 8,1 180792 5243589
/opt/hadoop-0.18.3_dumbo/lib/commons-net-1.4.1.jar
java 19625 mdesnoyer 12r REG 8,1 288534 5243598
/opt/hadoop-0.18.3_dumbo/lib/jets3t-0.6.0.jar
java 19625 mdesnoyer 13r REG 8,1 665638 5243592
/opt/hadoop-0.18.3_dumbo/lib/jetty-5.1.4.jar
java 19625 mdesnoyer 14r REG 8,1 121070 5243616
/opt/hadoop-0.18.3_dumbo/lib/junit-3.8.1.jar
java 19625 mdesnoyer 15r REG 8,1 9484 5243580
/opt/hadoop-0.18.3_dumbo/lib/kfs-0.1.3.jar
java 19625 mdesnoyer 16r REG 8,1 391834 5243599
/opt/hadoop-0.18.3_dumbo/lib/log4j-1.2.15.jar
java 19625 mdesnoyer 17r REG 8,1 65261 5243614
/opt/hadoop-0.18.3_dumbo/lib/oro-2.0.8.jar
java 19625 mdesnoyer 18r REG 8,1 97689 5243586
/opt/hadoop-0.18.3_dumbo/lib/servlet-api.jar
java 19625 mdesnoyer 19r REG 8,1 15345 5243588
/opt/hadoop-0.18.3_dumbo/lib/slf4j-api-1.4.3.jar
java 19625 mdesnoyer 20r REG 8,1 8601 5243613
/opt/hadoop-0.18.3_dumbo/lib/slf4j-log4j12-1.4.3.jar
java 19625 mdesnoyer 21r REG 8,1 15010 5243591
/opt/hadoop-0.18.3_dumbo/lib/xmlenc-0.52.jar
java 19625 mdesnoyer 22r REG 8,1 112341 5243583
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/commons-el.jar
java 19625 mdesnoyer 23r REG 8,1 351903 5243585
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-compiler.jar
java 19625 mdesnoyer 24r REG 8,1 105502 5243582
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jasper-runtime.jar
java 19625 mdesnoyer 25r REG 8,1 50493 5243584
/opt/hadoop-0.18.3_dumbo/lib/jetty-ext/jsp-api.jar
java 19625 mdesnoyer 26r CHR 1,8 1817 /dev/random
java 19625 mdesnoyer 27r CHR 1,9 1825 /dev/urandom
java 19625 mdesnoyer 28r REG 8,1 2145163 4138922
/home/mdesnoyer/src/graffiti0/summer09/hadoop/lda-mapreduce.jar
java 19625 mdesnoyer 29r REG 8,1 618950 6646646
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
java 19625 mdesnoyer 30r REG 8,1 1145980 6646075
/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
java 19625 mdesnoyer 31r FIFO 0,6 1344679 pipe
java 19625 mdesnoyer 32u sock 0,4 1344677 can't
identify protocol
java 19625 mdesnoyer 33w FIFO 0,6 1344679 pipe
java 19625 mdesnoyer 34u 0000 0,7 0 32 anon_inode
java 19625 mdesnoyer 35u unix 0xe280c700 1344680 socket
java 19625 mdesnoyer 36r IPv6 1348265 TCP
luna.corn.cnwk:42903->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 37w IPv6 1406254 TCP
luna.corn.cnwk:49999->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 38u IPv6 1407766 TCP
luna.corn.cnwk:51609->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 40u IPv6 1407979 TCP
luna.corn.cnwk:35437->c17-dw-dev50-hdfs-dn-n3.corn.com:50031 (ESTABLISHED)
java 19625 mdesnoyer 41u IPv6 1347947 TCP
luna.corn.cnwk:39687->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 42r IPv6 1347948 TCP
luna.corn.cnwk:35215->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 43w IPv6 1347949 TCP
luna.corn.cnwk:35216->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 44u IPv6 1347950 TCP
luna.corn.cnwk:35217->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 48u IPv6 1347976 TCP
luna.corn.cnwk:35218->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 49u IPv6 1347977 TCP
luna.corn.cnwk:35219->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 50u IPv6 1347978 TCP
luna.corn.cnwk:42880->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 51u IPv6 1348006 TCP
luna.corn.cnwk:35221->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 52u IPv6 1348007 TCP
luna.corn.cnwk:42882->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 53u IPv6 1348034 TCP
luna.corn.cnwk:35223->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 54u IPv6 1348035 TCP
luna.corn.cnwk:35224->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 55u IPv6 1348036 TCP
luna.corn.cnwk:42885->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 56u IPv6 1348064 TCP
luna.corn.cnwk:35226->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 57u IPv6 1348065 TCP
luna.corn.cnwk:42887->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 58u IPv6 1348092 TCP
luna.corn.cnwk:39701->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 59u IPv6 1348093 TCP
luna.corn.cnwk:35229->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 60u IPv6 1348094 TCP
luna.corn.cnwk:42890->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 61u IPv6 1348122 TCP
luna.corn.cnwk:55555->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 62u IPv6 1348123 TCP
luna.corn.cnwk:35232->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 63u IPv6 1348150 TCP
luna.corn.cnwk:39706->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 64u IPv6 1348151 TCP
luna.corn.cnwk:35234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 65u IPv6 1348178 TCP
luna.corn.cnwk:42895->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 66u IPv6 1348179 TCP
luna.corn.cnwk:55560->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 67u IPv6 1348188 TCP
luna.corn.cnwk:42897->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 68u IPv6 1348208 TCP
luna.corn.cnwk:42898->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 69u IPv6 1348209 TCP
luna.corn.cnwk:55563->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 70u IPv6 1348236 TCP
luna.corn.cnwk:55564->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 71u IPv6 1348237 TCP
luna.corn.cnwk:55565->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 72u IPv6 1348264 TCP
luna.corn.cnwk:35242->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 73u IPv6 1348266 TCP
luna.corn.cnwk:35244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 74u IPv6 1348293 TCP
luna.corn.cnwk:39718->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 75u IPv6 1348294 TCP
luna.corn.cnwk:35246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 76u IPv6 1348321 TCP
luna.corn.cnwk:42907->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 77u IPv6 1348322 TCP
luna.corn.cnwk:42908->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 78u IPv6 1348325 TCP
luna.corn.cnwk:35249->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 79u IPv6 1348351 TCP
luna.corn.cnwk:42910->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 80u IPv6 1348352 TCP
luna.corn.cnwk:39724->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 81u IPv6 1348379 TCP
luna.corn.cnwk:39725->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 82u IPv6 1348380 TCP
luna.corn.cnwk:55577->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 83u IPv6 1348407 TCP
luna.corn.cnwk:39727->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 84u IPv6 1348408 TCP
luna.corn.cnwk:39728->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 85u IPv6 1348409 TCP
luna.corn.cnwk:35256->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 86u IPv6 1348410 TCP
luna.corn.cnwk:34784->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 88r IPv6 1406224 TCP
luna.corn.cnwk:56589->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 89u IPv6 1406225 TCP
luna.corn.cnwk:45601->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 90r IPv6 1406226 TCP
luna.corn.cnwk:45602->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 91w IPv6 1406227 TCP
luna.corn.cnwk:45603->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 92u IPv6 1406228 TCP
luna.corn.cnwk:49973->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 94w IPv6 1407722 TCP
luna.corn.cnwk:48650->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 95u IPv6 1407734 TCP
luna.corn.cnwk:48652->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 96u IPv6 1406229 TCP
luna.corn.cnwk:54989->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 97u IPv6 1406230 TCP
luna.corn.cnwk:56595->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 98u IPv6 1406231 TCP
luna.corn.cnwk:49976->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 99u IPv6 1406232 TCP
luna.corn.cnwk:49977->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 100u IPv6 1406233 TCP
luna.corn.cnwk:45609->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 101u IPv6 1406234 TCP
luna.corn.cnwk:56599->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 102u IPv6 1406235 TCP
luna.corn.cnwk:45611->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 103u IPv6 1406236 TCP
luna.corn.cnwk:54996->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 104u IPv6 1406237 TCP
luna.corn.cnwk:45613->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 105u IPv6 1406238 TCP
luna.corn.cnwk:56603->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 106u IPv6 1406239 TCP
luna.corn.cnwk:56604->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 107u IPv6 1406240 TCP
luna.corn.cnwk:45616->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 108u IPv6 1406241 TCP
luna.corn.cnwk:45617->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 109u IPv6 1406242 TCP
luna.corn.cnwk:49987->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 110u IPv6 1406243 TCP
luna.corn.cnwk:49988->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 111u IPv6 1406244 TCP
luna.corn.cnwk:56609->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 112u IPv6 1406245 TCP
luna.corn.cnwk:45621->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 113u IPv6 1406246 TCP
luna.corn.cnwk:45622->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 114u IPv6 1406247 TCP
luna.corn.cnwk:55007->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 115u IPv6 1406248 TCP
luna.corn.cnwk:45624->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 116u IPv6 1406249 TCP
luna.corn.cnwk:55009->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 117u IPv6 1406250 TCP
luna.corn.cnwk:56615->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 118u IPv6 1406251 TCP
luna.corn.cnwk:55011->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 119u IPv6 1406252 TCP
luna.corn.cnwk:56617->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 120u IPv6 1406253 TCP
luna.corn.cnwk:55013->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 121u IPv6 1406255 TCP
luna.corn.cnwk:45631->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 122u IPv6 1406256 TCP
luna.corn.cnwk:56621->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 123u IPv6 1406257 TCP
luna.corn.cnwk:45633->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 124u IPv6 1406258 TCP
luna.corn.cnwk:50003->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 125u IPv6 1406259 TCP
luna.corn.cnwk:45635->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 126u IPv6 1406260 TCP
luna.corn.cnwk:56625->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 127u IPv6 1406261 TCP
luna.corn.cnwk:45637->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 128u IPv6 1406262 TCP
luna.corn.cnwk:55022->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 129u IPv6 1406263 TCP
luna.corn.cnwk:45639->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 130u IPv6 1406264 TCP
luna.corn.cnwk:56629->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 131u IPv6 1406265 TCP
luna.corn.cnwk:50010->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 132u IPv6 1406266 TCP
luna.corn.cnwk:56631->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 133u IPv6 1406267 TCP
luna.corn.cnwk:50012->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 134u IPv6 1407738 TCP
luna.corn.cnwk:42171->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 135u IPv6 1407739 TCP
luna.corn.cnwk:42172->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 136u IPv6 1407740 TCP
luna.corn.cnwk:48658->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 137u IPv6 1407741 TCP
luna.corn.cnwk:42174->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 138u IPv6 1407742 TCP
luna.corn.cnwk:51586->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 139u IPv6 1407743 TCP
luna.corn.cnwk:48661->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 143u IPv6 1407744 TCP
luna.corn.cnwk:51588->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 144u IPv6 1407745 TCP
luna.corn.cnwk:42178->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 145u IPv6 1407746 TCP
luna.corn.cnwk:54228->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 146u IPv6 1407747 TCP
luna.corn.cnwk:51591->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 147u IPv6 1407748 TCP
luna.corn.cnwk:48666->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 148u IPv6 1407749 TCP
luna.corn.cnwk:48667->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 149u IPv6 1407750 TCP
luna.corn.cnwk:42183->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 150u IPv6 1407751 TCP
luna.corn.cnwk:51595->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 151u IPv6 1407752 TCP
luna.corn.cnwk:54234->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 152u IPv6 1407753 TCP
luna.corn.cnwk:48671->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 153u IPv6 1407754 TCP
luna.corn.cnwk:48672->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 154u IPv6 1407755 TCP
luna.corn.cnwk:42188->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 155u IPv6 1407756 TCP
luna.corn.cnwk:54238->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 156u IPv6 1407757 TCP
luna.corn.cnwk:42190->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 157u IPv6 1407758 TCP
luna.corn.cnwk:51602->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 158u IPv6 1407759 TCP
luna.corn.cnwk:48677->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 159u IPv6 1407760 TCP
luna.corn.cnwk:42193->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 160u IPv6 1407761 TCP
luna.corn.cnwk:54243->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 161u IPv6 1407762 TCP
luna.corn.cnwk:54244->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 162u IPv6 1407763 TCP
luna.corn.cnwk:42196->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 163u IPv6 1407764 TCP
luna.corn.cnwk:54246->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 164u IPv6 1407767 TCP
luna.corn.cnwk:51610->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 165u IPv6 1407768 TCP
luna.corn.cnwk:42200->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 166u IPv6 1407769 TCP
luna.corn.cnwk:51612->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 167u IPv6 1407770 TCP
luna.corn.cnwk:51613->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 168u IPv6 1407771 TCP
luna.corn.cnwk:48688->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 169u IPv6 1407772 TCP
luna.corn.cnwk:42204->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 170u IPv6 1407773 TCP
luna.corn.cnwk:42205->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 171u IPv6 1407774 TCP
luna.corn.cnwk:51617->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 172u IPv6 1407775 TCP
luna.corn.cnwk:48692->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 173u IPv6 1407776 TCP
luna.corn.cnwk:42208->c17-dw-dev50-hdfs-dn-n1.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 174u IPv6 1407777 TCP
luna.corn.cnwk:54258->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 175u IPv6 1407778 TCP
luna.corn.cnwk:51621->c17-dw-dev50-hdfs-dn-n4.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 176u IPv6 1407779 TCP
luna.corn.cnwk:54260->c17-dw-dev50-hdfs-dn-n2.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 177u IPv6 1407780 TCP
luna.corn.cnwk:48697->c17-dw-dev50-hdfs-dn-n3.corn.com:50010 (CLOSE_WAIT)
java 19625 mdesnoyer 178r FIFO 0,6 1407966 pipe
java 19625 mdesnoyer 179w FIFO 0,6 1407966 pipe
java 19625 mdesnoyer 180u 0000 0,7 0 32 anon_inode
On Mon, Aug 3, 2009 at 6:10 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Attachments usually don't come through on ASF mailing lists.
>
> Also, Mahout trunk requires Hadoop 0.20.0
>
>
>
> On Aug 3, 2009, at 6:05 PM, Mark Desnoyer wrote:
>
> Hi,
>>
>> Has anybody had a problem running KMeans leaking TCP connections, which
>> eventually causes the OS to issue a "Too Many Files Open" error? In other
>> words, it looks like a file descriptor leak.
>>
>> I'm running a four machine cluster of Hadoop 18.3, with the current
>> version of Mahout from the trunk. Each k-means iteration is causing an extra
>> 40 or so TCP connections to be created that get orphaned in CLOSE_WAIT. I've
>> attached the output of lsof from the middle of a run.
>>
>> Any help would be appreciated!
>>
>> Cheers,
>> Mark
>>
>
>
>
Re: KMeans Leaked TCP Connections
Posted by Grant Ingersoll <gs...@apache.org>.
Attachments usually don't come through on ASF mailing lists.
Also, Mahout trunk requires Hadoop 0.20.0
On Aug 3, 2009, at 6:05 PM, Mark Desnoyer wrote:
> Hi,
>
> Has anybody had a problem running KMeans leaking TCP connections,
> which eventually causes the OS to issue a "Too Many Files Open"
> error? In other words, it looks like a file descriptor leak.
>
> I'm running a four machine cluster of Hadoop 18.3, with the current
> version of Mahout from the trunk. Each k-means iteration is causing
> an extra 40 or so TCP connections to be created that get orphaned in
> CLOSE_WAIT. I've attached the output of lsof from the middle of a run.
>
> Any help would be appreciated!
>
> Cheers,
> Mark