You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Bennie Schut <bs...@ebuddy.com> on 2010/02/15 15:09:02 UTC
Re: Hive Server Leaking File Descriptors?
Did this help? I'm running into a similar problem. slowly leaking
connections to 50010 and after a hive restart all is ok again.
Andy Kent wrote:
> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>
> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>
>
> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>
>
>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>
>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>
>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <ja...@gmail.com>> wrote:
>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>
>>
>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <An...@forward.co.uk>> wrote:
>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>
>>
>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>
>> Thanks for the response.
>>
>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>
>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>
>>
>> <SequenceFile.patch>
>>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
These all seems to be normal.
Can you attach the 133k file as an attachment? (At least all lines
with org.apache.hadoop.*, and java.*)
Zheng
On Tue, Feb 16, 2010 at 1:36 AM, Bennie Schut <bs...@ebuddy.com> wrote:
> That's 133k of data so perhaps this is enough?:
>
> 627: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 883: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 934: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 944: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> After a couple more queries:
>
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 991: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> some more stuff
> cat jmap.txt | grep hadoop.fs
> 535: 8 192
> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
> 539: 8 192 org.apache.hadoop.fs.permission.FsAction
> 611: 8 128 org.apache.hadoop.fs.Path
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 721: 4 96
> org.apache.hadoop.fs.permission.FsPermission$2
> 760: 2 96
> [Lorg.apache.hadoop.fs.permission.FsAction;
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1213: 1 24
> org.apache.hadoop.fs.permission.FsPermission
> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
> 1413: 1 16
> org.apache.hadoop.fs.ChecksumFileSystem$1
> 1523: 1 16
> org.apache.hadoop.fs.permission.FsPermission$1
> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
>
>
> Currently:
> lsof | grep "50010 (ESTABLISHED)" | wc -l
> 453
>
> Zheng Shao wrote:
>>
>> Thanks for the quick reply. It seems the number of threads are normal.
>>
>> Can you do "jmap -histo:live" as well to find out the number of
>> DFSClient, FileSystem, etc?
>>
>>
>> On 2/16/10, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>>>
>>> jstack on the Hive process:
>>>
>>> 2010-02-16
>>> 09:30:47
>>>
>>>
>>>
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>> mode):
>>>
>>>
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>>> condition [0x000000004238d000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
>>> condition [0x0000000042690000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
>>> on condition [0x0000000042791000]
>>> java.lang.Thread.State: TIMED_WAITING
>>> (sleeping)
>>> at java.lang.Thread.$$YJP$$sleep(Native
>>> Method)
>>> at
>>> java.lang.Thread.sleep(Thread.java)
>>>
>>>
>>> at
>>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
>>> condition [0x000000004258f000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
>>> condition [0x000000004248e000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
>>> condition [0x0000000040c7e000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
>>> nid=0x179d runnable [0x000000004228c000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>> at java.net.SocketInputStream.socketRead0(Native
>>> Method)
>>> at
>>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>
>>>
>>> - locked <0x00007fbf10922580> (a
>>> java.io.BufferedInputStream)
>>> at
>>> java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>
>>>
>>> at
>>> java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>
>>>
>>> at
>>> com.yourkit.runtime.Packet.readHeader(Packet.java:40)
>>>
>>>
>>> at
>>>
>>> com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
>>>
>>>
>>> at
>>> com.yourkit.runtime.Core$4.run(Core.java:525)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
>>> waiting on condition [0x0000000040d7f000]
>>> java.lang.Thread.State: TIMED_WAITING
>>> (sleeping)
>>> at java.lang.Thread.$$YJP$$sleep(Native
>>> Method)
>>> at
>>> java.lang.Thread.sleep(Thread.java)
>>>
>>>
>>> at
>>>
>>> com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
>>>
>>>
>>>
>>> "Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
>>> nid=0x1795 runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
>>> Object.wait() [0x000000004119d000]
>>> java.lang.Thread.State: WAITING (on object
>>> monitor)
>>> at java.lang.Object.wait(Native
>>> Method)
>>> - waiting on <0x00007fbea52c6790> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>>
>>>
>>> - locked <0x00007fbea52c6790> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>>
>>>
>>> at
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>
>>>
>>>
>>> "Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
>>> Object.wait() [0x000000004208a000]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> - waiting on <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>> at java.lang.Object.wait(Object.java:485)
>>> at
>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> - locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>>
>>> "main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
>>> [0x00000000414e3000]
>>> java.lang.Thread.State: RUNNABLE
>>> at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>>> - locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>>
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>>> at
>>>
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
>>> at
>>>
>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>> at
>>>
>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
>>> at
>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>> "VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
>>>
>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
>>> nid=0x1782 runnable
>>>
>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
>>> nid=0x1783 runnable
>>>
>>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
>>> nid=0x1784 runnable
>>>
>>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
>>> nid=0x1785 runnable
>>>
>>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
>>> nid=0x1786 runnable
>>>
>>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
>>> nid=0x1787 runnable
>>>
>>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
>>> nid=0x1788 runnable
>>>
>>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
>>> nid=0x1789 runnable
>>>
>>> "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
>>> nid=0x178a runnable
>>>
>>> "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
>>> nid=0x178b runnable
>>>
>>> "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
>>> nid=0x178c runnable
>>>
>>> "GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
>>> nid=0x178d runnable
>>>
>>> "GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
>>> nid=0x178e runnable
>>>
>>> "VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
>>> waiting on condition
>>>
>>> JNI global references: 375194
>>>
>>>
>>>
>>> Zheng Shao wrote:
>>>
>>>>
>>>> It seems that Hive server is leaking hdfs handles.
>>>> Not sure if hadoop has a command to show open hdfs handles.
>>>>
>>>> Can you also do a jstack of the Hive Server process?
>>>> First, "ps -elf | grep java" to find the hive server process id, and
>>>> the jdk directory
>>>> Second, "<jdk directory>/bin/jstack <pid>".
>>>>
>>>> This will tell us if there are clients of the Hive Server that didn't
>>>> close their connections yet.
>>>>
>>>> Zheng
>>>>
>>>> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> That's the datanode port. So Hdfs. Any trick to see what is opened on
>>>>> hdfs?
>>>>> like an lsof for hdfs :)
>>>>>
>>>>> Zheng Shao wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> What's running at victoria.ebuddy.com:50010 ?
>>>>>>
>>>>>> Zhemg
>>>>>>
>>>>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I think it's lines like this:
>>>>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>>>>
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>>
>>>>>>> We have the same.
>>>>>>> Currently no queries running and many of those connections open.
>>>>>>> After
>>>>>>> running some queries a few more are added to the list.
>>>>>>>
>>>>>>> java 5888 dwh 389u IPv4 1020083623
>>>>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 390u IPv4 1020083626
>>>>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 391u IPv4 1020083637
>>>>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 392u IPv4 1020083641
>>>>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 393u IPv4 1020083668
>>>>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 394u IPv4 1020083675
>>>>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 395u IPv4 1020083678
>>>>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 396u IPv4 1020083690
>>>>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 397u IPv4 1020083693
>>>>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 398u IPv4 1020083703
>>>>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 399u IPv4 1020083706
>>>>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 400u IPv4 1020083713
>>>>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 401u IPv4 1020083716
>>>>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 402u IPv4 1020083728
>>>>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 403u IPv4 1020083731
>>>>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 404u IPv4 1020083740
>>>>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 405u IPv4 1020083743
>>>>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 406u IPv4 1020083753
>>>>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 407u IPv4 1020083756
>>>>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 408u IPv4 1020083769
>>>>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 409u IPv4 1020083772
>>>>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 410u IPv4 1020083781
>>>>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 411u IPv4 1020083784
>>>>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 412u IPv4 1020083792
>>>>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 413u IPv4 1020083795
>>>>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 414u IPv4 1020083808
>>>>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 415u IPv4 1020083811
>>>>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 416u IPv4 1020083820
>>>>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 417u IPv4 1020083823
>>>>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 418u IPv4 1020083831
>>>>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 419u IPv4 1020083834
>>>>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 420u IPv4 1020083846
>>>>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 421u IPv4 1020083849
>>>>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 422u IPv4 1020083865
>>>>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 423u IPv4 1020083862
>>>>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 424u IPv4 1020083877
>>>>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 425u IPv4 1020083880
>>>>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 426u IPv4 1020083888
>>>>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 427u IPv4 1020083896
>>>>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 433u IPv4 1020083914
>>>>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 434u IPv4 1020083916
>>>>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 435u IPv4 1020083926
>>>>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 436u IPv4 1020083929
>>>>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 437u IPv4 1020084012
>>>>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 438u IPv4 1020083947
>>>>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 439u IPv4 1020083975
>>>>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 443u IPv4 1020083950
>>>>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 444u IPv4 1020083959
>>>>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 445u IPv4 1020083962
>>>>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 446u IPv4 1020083977
>>>>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 447u IPv4 1020083984
>>>>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 448u IPv4 1020088663
>>>>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 449u IPv4 1020083991
>>>>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 450u IPv4 1020088678
>>>>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 451u IPv4 1020088692
>>>>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 452u IPv4 1020088694
>>>>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 453u IPv4 1020092163
>>>>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 5888 dwh 454u IPv4 1020092165
>>>>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010
>>>>>>> (ESTABLISHED)
>>>>>>>
>>>>>>> Andy Kent wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I have included the output from lsof, Unfortunately though I only
>>>>>>>> restarted the server a few mins ago and so it doesn't really
>>>>>>>> illustrate
>>>>>>>> the
>>>>>>>> problem. I'll try to post back with the same command tomorrow so
>>>>>>>> that
>>>>>>>> we
>>>>>>>> can
>>>>>>>> diff between the two.
>>>>>>>>
>>>>>>>> Thanks, Andy.
>>>>>>>>
>>>>>>>> java 16908 hadoop cwd DIR 0,17 240
>>>>>>>> 7037865 /var/run/hadoop-0.20
>>>>>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>>>>>> 2 /
>>>>>>>> java 16908 hadoop txt REG 8,1 50810
>>>>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>>>>> java 16908 hadoop mem REG 8,1 127296
>>>>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>>>>> java 16908 hadoop mem REG 8,1 69472
>>>>>>>> 3735429
>>>>>>>>
>>>>>>>>
>>>>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>>>>> java 16908 hadoop mem REG 8,1 44257
>>>>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>>>>> java 16908 hadoop mem REG 8,1 80760
>>>>>>>> 1164352 /lib/libresolv-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 22856
>>>>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 1145980
>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38105
>>>>>>>> 3555614
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>>>>> java 16908 hadoop mem REG 8,1 111901
>>>>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>>>>> java 16908 hadoop mem REG 8,1 392124
>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 229928
>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 724225
>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 169666
>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 45946
>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 87325
>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 179508
>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 20202
>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 95488
>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 390114
>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 500917
>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 31931
>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 19349
>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1798010
>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 11412
>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 14099
>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 12641
>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 7092
>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 2446767
>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1154170
>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 176174
>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1923209
>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 261809
>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 575389
>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 258337
>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 43035
>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 105209
>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 134910
>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1024681
>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 15010
>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 8601
>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 15345
>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 132368
>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 65261
>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 540852
>>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 73374
>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 11428
>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 198945
>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 163121
>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 516429
>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 321806
>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 76698
>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 405086
>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 37210
>>>>>>>> 6038622
>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 18684
>>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 37157
>>>>>>>> 3721359
>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 3566844
>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 180792
>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 279781
>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 2771746
>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 12588713
>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 25700
>>>>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>>>>> java 16908 hadoop mem REG 8,1 254076
>>>>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>>>>> java 16908 hadoop mem REG 8,1 51669471
>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 92117
>>>>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>>>>> java 16908 hadoop mem REG 8,1 47528
>>>>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 43480
>>>>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 35632
>>>>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 55850
>>>>>>>> 3555581
>>>>>>>>
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>>>>> java 16908 hadoop mem REG 8,1 93080
>>>>>>>> 1164343 /lib/libnsl-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 234713
>>>>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>>>>> java 16908 hadoop mem REG 8,1 66580
>>>>>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>>>>> java 16908 hadoop mem REG 8,1 35784
>>>>>>>> 1164353 /lib/librt-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 526560
>>>>>>>> 1164341 /lib/libm-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 10987257
>>>>>>>> 3555599
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>>>>> java 16908 hadoop mem REG 8,1 1436976
>>>>>>>> 1164337 /lib/libc-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 14624
>>>>>>>> 1164340 /lib/libdl-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 130224
>>>>>>>> 1164351 /lib/libpthread-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 127480
>>>>>>>> 1164334 /lib/ld-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 618950
>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 112341
>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 32768
>>>>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>>>>> java 16908 hadoop mem REG 8,1 48335
>>>>>>>> 3555565
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>>>>> java 16908 hadoop mem REG 8,1 41123
>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>> java 16908 hadoop 0u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 1u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 2u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>> java 16908 hadoop 6r REG 8,1 41123
>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>> java 16908 hadoop 7r REG 8,1 46725
>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop 8r REG 8,1 112341
>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>> java 16908 hadoop 9r REG 8,1 279781
>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>> java 16908 hadoop 10r REG 8,1 38015
>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop 11r REG 8,1 26202
>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop 12r REG 8,1 180792
>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>> java 16908 hadoop 14r REG 8,1 37157
>>>>>>>> 3721359
>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>> java 16908 hadoop 15r REG 8,1 18684
>>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>> java 16908 hadoop 16r REG 8,1 37210
>>>>>>>> 6038622
>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>> java 16908 hadoop 17r REG 8,1 706710
>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>> java 16908 hadoop 18r REG 8,1 706710
>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>> java 16908 hadoop 19r REG 8,1 405086
>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>> java 16908 hadoop 20r REG 8,1 76698
>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>> java 16908 hadoop 21r REG 8,1 321806
>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>> java 16908 hadoop 22r REG 8,1 516429
>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>> java 16908 hadoop 23r REG 8,1 163121
>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>> java 16908 hadoop 24r REG 8,1 121070
>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop 25r REG 8,1 198945
>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>> java 16908 hadoop 26r REG 8,1 11428
>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>> java 16908 hadoop 27r REG 8,1 114342
>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>> java 16908 hadoop 28r REG 8,1 73374
>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>> java 16908 hadoop 29r REG 8,1 391834
>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop 30r REG 8,1 540852
>>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>> java 16908 hadoop 31r REG 8,1 65261
>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>> java 16908 hadoop 32r REG 8,1 132368
>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>> java 16908 hadoop 33r REG 8,1 15345
>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>> java 16908 hadoop 34r REG 8,1 8601
>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>> java 16908 hadoop 35r REG 8,1 15010
>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>> java 16908 hadoop 37r REG 8,1 134910
>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>> java 16908 hadoop 38r REG 8,1 105209
>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>> java 16908 hadoop 39r REG 8,1 43035
>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>> java 16908 hadoop 40r REG 8,1 258337
>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop 41r REG 8,1 46725
>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop 42r REG 8,1 575389
>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>> java 16908 hadoop 43r REG 8,1 261809
>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>> java 16908 hadoop 44r REG 8,1 38015
>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop 45r REG 8,1 26202
>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>> java 16908 hadoop 47r REG 8,1 176174
>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>> java 16908 hadoop 50r REG 8,1 7092
>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>> java 16908 hadoop 51r REG 8,1 12641
>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>> java 16908 hadoop 52r REG 8,1 14099
>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>> java 16908 hadoop 53r REG 8,1 11412
>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>> java 16908 hadoop 55r REG 8,1 19349
>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>> java 16908 hadoop 56r REG 8,1 31931
>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>> java 16908 hadoop 57r REG 8,1 500917
>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>> java 16908 hadoop 58r REG 8,1 390114
>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>> java 16908 hadoop 59r REG 8,1 95488
>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>> java 16908 hadoop 60r REG 8,1 20202
>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>> java 16908 hadoop 61r REG 8,1 179508
>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop 62r REG 8,1 87325
>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>> java 16908 hadoop 63r REG 8,1 45946
>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>> java 16908 hadoop 64r REG 8,1 121070
>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop 65r REG 8,1 114342
>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>> java 16908 hadoop 66r REG 8,1 169666
>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>> java 16908 hadoop 67r REG 8,1 391834
>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop 68r REG 8,1 724225
>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>> java 16908 hadoop 69r REG 8,1 229928
>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>> java 16908 hadoop 70r REG 8,1 392124
>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>> java 16908 hadoop 71r CHR 1,8
>>>>>>>> 1818 /dev/random
>>>>>>>> java 16908 hadoop 72r CHR 1,9
>>>>>>>> 1824 /dev/urandom
>>>>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>>>>> TCP *:webmin (LISTEN)
>>>>>>>> java 16908 hadoop 74u sock 0,4
>>>>>>>> 153142739 can't identify protocol
>>>>>>>> java 16908 hadoop 76w REG 8,1 6867
>>>>>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 81w REG 8,1 15491
>>>>>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 83r REG 8,1 618950
>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>>>>> 153143422 socket
>>>>>>>> java 16908 hadoop 86w REG 8,1 62510
>>>>>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>>>>> java 16908 hadoop 87w REG 8,1 960
>>>>>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 113w REG 8,1 11562
>>>>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>>>>> 154318314 pipe
>>>>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>>>>> 154318314 pipe
>>>>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 249w REG 8,1 15230
>>>>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 253u 0000 0,6 0
>>>>>>>> 493 /anon_inode:[eventpoll]
>>>>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>>>>> TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345"
>>>>>>>>> where
>>>>>>>>> 12345 is the process id of the hive server?
>>>>>>>>> We should be able to see the names of the files that are open.
>>>>>>>>>
>>>>>>>>> Zheng
>>>>>>>>>
>>>>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent
>>>>>>>>> <An...@forward.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nope, no luck so far.
>>>>>>>>>>
>>>>>>>>>> We have upped the number of file descriptors and are having to
>>>>>>>>>> restart
>>>>>>>>>> hive every week or so :(
>>>>>>>>>>
>>>>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>>>>
>>>>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>>>>
>>>>>>>>>>> Andy Kent wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I can give try and give it a go. I'm not convinced though as we
>>>>>>>>>>>> are
>>>>>>>>>>>> working with CSV files and don't touch sequence files at all at
>>>>>>>>>>>> the
>>>>>>>>>>>> moment.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133
>>>>>>>>>>>> and
>>>>>>>>>>>> Hive 0.40
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Actually, we had an issue with this, it was a bug in
>>>>>>>>>>>>> SequenceFile
>>>>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>>>>> filehandle
>>>>>>>>>>>>> open and never close it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get
>>>>>>>>>>>>> some
>>>>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>>>>>> this to
>>>>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be
>>>>>>>>>>>>> a
>>>>>>>>>>>>> combination.. maybe if you're doing queries and then closing
>>>>>>>>>>>>> your
>>>>>>>>>>>>> thrift
>>>>>>>>>>>>> connection before reading all results, Hive doesn't know what
>>>>>>>>>>>>> to
>>>>>>>>>>>>> do
>>>>>>>>>>>>> and
>>>>>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>>>>>> they might
>>>>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>>>>> <An...@forward.co.uk>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's
>>>>>>>>>>>>>> connecting
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?)
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's what I was suspecting. I have triple checked and our
>>>>>>>>>>>>> Ruby
>>>>>>>>>>>>> code
>>>>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll try running some different queries and see if I can suss
>>>>>>>>>>>>> out
>>>>>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>>>>>> should post
>>>>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>>>>> haven't noticed
>>>>>>>>>>>>> this?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Yours,
>>>>>>>>> Zheng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Yours,
>>>>>> Zheng
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>>
>>>
>>>
>>
>> --
>> Sent from my mobile device
>>
>> Yours,
>> Zheng
>>
>
>
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
fyi.. just tested this and I see no more leaks. Thanks.
Zheng Shao wrote:
> https://issues.apache.org/jira/browse/HIVE-1181 is committed.
> Thanks to Yonqqiang.
>
> We need to "-hiveconf hive.fileformat.check=false" when starting
> HiveServer to get rid of the extra connections.
>
> We still need to fix MAPREDUCE-1504 so that we can re-enable fileformat check.
>
> Zheng
>
> On Thu, Feb 18, 2010 at 11:47 PM, Bennie Schut <bs...@ebuddy.com> wrote:
>
>> That's some great news. Thanks.
>>
>> Zheng Shao wrote:
>>
>>> This is actually a bug in MAPREDUCE-1504, but we will try to find a
>>> workaround.
>>> https://issues.apache.org/jira/browse/HIVE-1181
>>>
>>> Given that release 0.5.0 is much wanted right now, I don't think we
>>> want to wait purely for 0.5.0 since the ultimate fix should come from
>>> Hadoop.
>>> We will definitely get HIVE-1181 for branch 0.5.
>>>
>>> Zheng
>>>
>>> ---------- Forwarded message ----------
>>> From: Andy Kent <An...@forward.co.uk>
>>> Date: Thu, Feb 18, 2010 at 3:17 PM
>>> Subject: Re: Hive Server Leaking File Descriptors?
>>> To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
>>>
>>>
>>>
>>> On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>>>
>>>
>>>
>>>>> I've tried to look into it a bit more and it seems to happen on
>>>>> "load data
>>>>> inpath"
>>>>>
>>>>>
>>> This is inline with what we have been seeing as we do around 200 load
>>> data statements per day and leak approx the same number of file
>>> descriptors.
>>>
>>> Is there any chance this fix will make it into the 0.5 release?
>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
https://issues.apache.org/jira/browse/HIVE-1181 is committed.
Thanks to Yonqqiang.
We need to "-hiveconf hive.fileformat.check=false" when starting
HiveServer to get rid of the extra connections.
We still need to fix MAPREDUCE-1504 so that we can re-enable fileformat check.
Zheng
On Thu, Feb 18, 2010 at 11:47 PM, Bennie Schut <bs...@ebuddy.com> wrote:
> That's some great news. Thanks.
>
> Zheng Shao wrote:
>>
>> This is actually a bug in MAPREDUCE-1504, but we will try to find a
>> workaround.
>> https://issues.apache.org/jira/browse/HIVE-1181
>>
>> Given that release 0.5.0 is much wanted right now, I don't think we
>> want to wait purely for 0.5.0 since the ultimate fix should come from
>> Hadoop.
>> We will definitely get HIVE-1181 for branch 0.5.
>>
>> Zheng
>>
>> ---------- Forwarded message ----------
>> From: Andy Kent <An...@forward.co.uk>
>> Date: Thu, Feb 18, 2010 at 3:17 PM
>> Subject: Re: Hive Server Leaking File Descriptors?
>> To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
>>
>>
>>
>> On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>>
>>
>>>>
>>>> I've tried to look into it a bit more and it seems to happen on
>>>> "load data
>>>> inpath"
>>>>
>>
>> This is inline with what we have been seeing as we do around 200 load
>> data statements per day and leak approx the same number of file
>> descriptors.
>>
>> Is there any chance this fix will make it into the 0.5 release?
>>
>>
>>
>>
>
>
--
Yours,
Zheng
RE: Hive Server Leaking File Descriptors?
Posted by Andy Kent <An...@forward.co.uk>.
This has made my day.
Thanks for working through this Bennie and Zheng.
Andy.
________________________________________
From: Bennie Schut [bschut@ebuddy.com]
Sent: 19 February 2010 07:47
To: hive-user@hadoop.apache.org
Subject: Re: Hive Server Leaking File Descriptors?
That's some great news. Thanks.
Zheng Shao wrote:
> This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
> https://issues.apache.org/jira/browse/HIVE-1181
>
> Given that release 0.5.0 is much wanted right now, I don't think we
> want to wait purely for 0.5.0 since the ultimate fix should come from
> Hadoop.
> We will definitely get HIVE-1181 for branch 0.5.
>
> Zheng
>
> ---------- Forwarded message ----------
> From: Andy Kent <An...@forward.co.uk>
> Date: Thu, Feb 18, 2010 at 3:17 PM
> Subject: Re: Hive Server Leaking File Descriptors?
> To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
>
>
>
> On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>
>
>>> I've tried to look into it a bit more and it seems to happen on
>>> "load data
>>> inpath"
>>>
>
> This is inline with what we have been seeing as we do around 200 load
> data statements per day and leak approx the same number of file
> descriptors.
>
> Is there any chance this fix will make it into the 0.5 release?
>
>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
That's some great news. Thanks.
Zheng Shao wrote:
> This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
> https://issues.apache.org/jira/browse/HIVE-1181
>
> Given that release 0.5.0 is much wanted right now, I don't think we
> want to wait purely for 0.5.0 since the ultimate fix should come from
> Hadoop.
> We will definitely get HIVE-1181 for branch 0.5.
>
> Zheng
>
> ---------- Forwarded message ----------
> From: Andy Kent <An...@forward.co.uk>
> Date: Thu, Feb 18, 2010 at 3:17 PM
> Subject: Re: Hive Server Leaking File Descriptors?
> To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
>
>
>
> On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>
>
>>> I've tried to look into it a bit more and it seems to happen on
>>> "load data
>>> inpath"
>>>
>
> This is inline with what we have been seeing as we do around 200 load
> data statements per day and leak approx the same number of file
> descriptors.
>
> Is there any chance this fix will make it into the 0.5 release?
>
>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
https://issues.apache.org/jira/browse/HIVE-1181
Given that release 0.5.0 is much wanted right now, I don't think we
want to wait purely for 0.5.0 since the ultimate fix should come from
Hadoop.
We will definitely get HIVE-1181 for branch 0.5.
Zheng
---------- Forwarded message ----------
From: Andy Kent <An...@forward.co.uk>
Date: Thu, Feb 18, 2010 at 3:17 PM
Subject: Re: Hive Server Leaking File Descriptors?
To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>> I've tried to look into it a bit more and it seems to happen on
>> "load data
>> inpath"
This is inline with what we have been seeing as we do around 200 load
data statements per day and leak approx the same number of file
descriptors.
Is there any chance this fix will make it into the 0.5 release?
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Andy Kent <An...@forward.co.uk>.
On 18 Feb 2010, at 20:29, "Zheng Shao" <zs...@gmail.com> wrote:
>> I've tried to look into it a bit more and it seems to happen on
>> "load data
>> inpath"
This is inline with what we have been seeing as we do around 200 load
data statements per day and leak approx the same number of file
descriptors.
Is there any chance this fix will make it into the 0.5 release?
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
Bennie,
Thanks for the detailed information! I think we've found the bug.
I just opened this:
https://issues.apache.org/jira/browse/MAPREDUCE-1504
Hive does check of the file format (to be consistent with the file
format declared in the table) when users load data files into Hive.
In order to verify that a file is a text file, Hive uses
SequenceFile.Reader class to open the file. If an IOException is
thrown, we know it's not a SequenceFile.
However, when an IOException is thrown, the
SequenceFile.Reader.close() is not called. I believe that is leaking
resources.
Please take a look at the JIRA above and put in any comments.
Zheng
On Thu, Feb 18, 2010 at 6:27 AM, Bennie Schut <bs...@ebuddy.com> wrote:
> I've tried to look into it a bit more and it seems to happen on "load data
> inpath"
> LOAD DATA INPATH '/user/hive/warehouse/chatsessions_2010-02-01.csv' INTO
> TABLE chatsessions_load;
>
> And not on things like
> select count(1) from chatsessions_load;
>
>
>
> Bennie Schut wrote:
>>
>> That's 133k of data so perhaps this is enough?:
>>
>> 627: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
>> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 883: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 934: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 944: 1 48
>> org.apache.hadoop.hdfs.DistributedFileSystem
>> 1082: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>> After a couple more queries:
>>
>> 671: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
>> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 931: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 981: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 991: 1 48
>> org.apache.hadoop.hdfs.DistributedFileSystem
>> 1132: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>> some more stuff
>> cat jmap.txt | grep hadoop.fs
>> 535: 8 192
>> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
>> 539: 8 192
>> org.apache.hadoop.fs.permission.FsAction
>> 611: 8 128 org.apache.hadoop.fs.Path
>> 671: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 721: 4 96
>> org.apache.hadoop.fs.permission.FsPermission$2
>> 760: 2 96
>> [Lorg.apache.hadoop.fs.permission.FsAction;
>> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 931: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 981: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 1132: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1213: 1 24
>> org.apache.hadoop.fs.permission.FsPermission
>> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
>> 1413: 1 16
>> org.apache.hadoop.fs.ChecksumFileSystem$1
>> 1523: 1 16
>> org.apache.hadoop.fs.permission.FsPermission$1
>> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
>> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>>
>>
>> Currently:
>> lsof | grep "50010 (ESTABLISHED)" | wc -l
>> 453
>>
>> Zheng Shao wrote:
>>
>>>
>>> Thanks for the quick reply. It seems the number of threads are normal.
>>>
>>> Can you do "jmap -histo:live" as well to find out the number of
>>> DFSClient, FileSystem, etc?
>>>
>>>
>>> On 2/16/10, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>>
>>>>
>>>> jstack on the Hive process:
>>>>
>>>> 2010-02-16
>>>> 09:30:47
>>>>
>>>>
>>>>
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>>> mode):
>>>>
>>>>
>>>>
>>>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>>
>>>>
>>>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>>>> condition [0x000000004238d000]
>>>> java.lang.Thread.State: WAITING
>>>> (parking)
>>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>>> Method)
>>>> - parking to wait for <0x00007fbea529a0d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> sun.misc.Unsafe.park(Unsafe.java)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
>>>> condition [0x0000000042690000]
>>>> java.lang.Thread.State: WAITING
>>>> (parking)
>>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>>> Method)
>>>> - parking to wait for <0x00007fbea529a0d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> sun.misc.Unsafe.park(Unsafe.java)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
>>>> on condition [0x0000000042791000]
>>>> java.lang.Thread.State: TIMED_WAITING
>>>> (sleeping)
>>>> at java.lang.Thread.$$YJP$$sleep(Native
>>>> Method)
>>>> at
>>>> java.lang.Thread.sleep(Thread.java)
>>>>
>>>>
>>>> at
>>>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
>>>> condition [0x000000004258f000]
>>>> java.lang.Thread.State: WAITING
>>>> (parking)
>>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>>> Method)
>>>> - parking to wait for <0x00007fbea529a0d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> sun.misc.Unsafe.park(Unsafe.java)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
>>>> condition [0x000000004248e000]
>>>> java.lang.Thread.State: WAITING
>>>> (parking)
>>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>>> Method)
>>>> - parking to wait for <0x00007fbea529a0d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> sun.misc.Unsafe.park(Unsafe.java)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
>>>> condition [0x0000000040c7e000]
>>>> java.lang.Thread.State: WAITING
>>>> (parking)
>>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>>> Method)
>>>> - parking to wait for <0x00007fbea529a0d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> sun.misc.Unsafe.park(Unsafe.java)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>>
>>>>
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
>>>> nid=0x179d runnable [0x000000004228c000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>>
>>>> at java.net.SocketInputStream.socketRead0(Native
>>>> Method)
>>>> at
>>>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>
>>>>
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>
>>>>
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>
>>>>
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>
>>>>
>>>> - locked <0x00007fbf10922580> (a
>>>> java.io.BufferedInputStream)
>>>> at
>>>> java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>>
>>>>
>>>> at
>>>> java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>
>>>>
>>>> at
>>>> com.yourkit.runtime.Packet.readHeader(Packet.java:40)
>>>>
>>>>
>>>> at
>>>>
>>>> com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
>>>>
>>>>
>>>> at
>>>> com.yourkit.runtime.Core$4.run(Core.java:525)
>>>>
>>>>
>>>> at
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>>
>>>> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
>>>> waiting on condition [0x0000000040d7f000]
>>>> java.lang.Thread.State: TIMED_WAITING
>>>> (sleeping)
>>>> at java.lang.Thread.$$YJP$$sleep(Native
>>>> Method)
>>>> at
>>>> java.lang.Thread.sleep(Thread.java)
>>>>
>>>>
>>>> at
>>>>
>>>> com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
>>>>
>>>>
>>>>
>>>> "Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
>>>> runnable [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>> "CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>>
>>>>
>>>> "CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>>
>>>>
>>>> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
>>>> nid=0x1795 runnable [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>>
>>>>
>>>> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
>>>> runnable [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>> "Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
>>>> runnable [0x0000000000000000]
>>>> java.lang.Thread.State:
>>>> RUNNABLE
>>>>
>>>> "Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
>>>> Object.wait() [0x000000004119d000]
>>>> java.lang.Thread.State: WAITING (on object
>>>> monitor)
>>>> at java.lang.Object.wait(Native
>>>> Method)
>>>> - waiting on <0x00007fbea52c6790> (a
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>> at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>>>
>>>>
>>>> - locked <0x00007fbea52c6790> (a
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>> at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>>>
>>>>
>>>> at
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>
>>>>
>>>>
>>>> "Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
>>>> Object.wait() [0x000000004208a000]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x00007fbea52c6450> (a
>>>> java.lang.ref.Reference$Lock)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>> - locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>>>
>>>> "main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
>>>> [0x00000000414e3000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>>>> - locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>>> at
>>>>
>>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
>>>> at
>>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>
>>>> "VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
>>>>
>>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
>>>> nid=0x1782 runnable
>>>>
>>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
>>>> nid=0x1783 runnable
>>>>
>>>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
>>>> nid=0x1784 runnable
>>>>
>>>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
>>>> nid=0x1785 runnable
>>>>
>>>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
>>>> nid=0x1786 runnable
>>>>
>>>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
>>>> nid=0x1787 runnable
>>>>
>>>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
>>>> nid=0x1788 runnable
>>>>
>>>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
>>>> nid=0x1789 runnable
>>>>
>>>> "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
>>>> nid=0x178a runnable
>>>>
>>>> "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
>>>> nid=0x178b runnable
>>>>
>>>> "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
>>>> nid=0x178c runnable
>>>>
>>>> "GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
>>>> nid=0x178d runnable
>>>>
>>>> "GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
>>>> nid=0x178e runnable
>>>>
>>>> "VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
>>>> waiting on condition
>>>>
>>>> JNI global references: 375194
>>>>
>>>>
>>>>
>>>> Zheng Shao wrote:
>>>>
>>>>
>>>>>
>>>>> It seems that Hive server is leaking hdfs handles.
>>>>> Not sure if hadoop has a command to show open hdfs handles.
>>>>>
>>>>> Can you also do a jstack of the Hive Server process?
>>>>> First, "ps -elf | grep java" to find the hive server process id, and
>>>>> the jdk directory
>>>>> Second, "<jdk directory>/bin/jstack <pid>".
>>>>>
>>>>> This will tell us if there are clients of the Hive Server that didn't
>>>>> close their connections yet.
>>>>>
>>>>> Zheng
>>>>>
>>>>> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> That's the datanode port. So Hdfs. Any trick to see what is opened on
>>>>>> hdfs?
>>>>>> like an lsof for hdfs :)
>>>>>>
>>>>>> Zheng Shao wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> What's running at victoria.ebuddy.com:50010 ?
>>>>>>>
>>>>>>> Zhemg
>>>>>>>
>>>>>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I think it's lines like this:
>>>>>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>>>>>
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>>
>>>>>>>> We have the same.
>>>>>>>> Currently no queries running and many of those connections open.
>>>>>>>> After
>>>>>>>> running some queries a few more are added to the list.
>>>>>>>>
>>>>>>>> java 5888 dwh 389u IPv4 1020083623
>>>>>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 390u IPv4 1020083626
>>>>>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 391u IPv4 1020083637
>>>>>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 392u IPv4 1020083641
>>>>>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 393u IPv4 1020083668
>>>>>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 394u IPv4 1020083675
>>>>>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 395u IPv4 1020083678
>>>>>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 396u IPv4 1020083690
>>>>>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 397u IPv4 1020083693
>>>>>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 398u IPv4 1020083703
>>>>>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 399u IPv4 1020083706
>>>>>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 400u IPv4 1020083713
>>>>>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 401u IPv4 1020083716
>>>>>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 402u IPv4 1020083728
>>>>>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 403u IPv4 1020083731
>>>>>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 404u IPv4 1020083740
>>>>>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 405u IPv4 1020083743
>>>>>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 406u IPv4 1020083753
>>>>>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 407u IPv4 1020083756
>>>>>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 408u IPv4 1020083769
>>>>>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 409u IPv4 1020083772
>>>>>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 410u IPv4 1020083781
>>>>>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 411u IPv4 1020083784
>>>>>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 412u IPv4 1020083792
>>>>>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 413u IPv4 1020083795
>>>>>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 414u IPv4 1020083808
>>>>>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 415u IPv4 1020083811
>>>>>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 416u IPv4 1020083820
>>>>>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 417u IPv4 1020083823
>>>>>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 418u IPv4 1020083831
>>>>>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 419u IPv4 1020083834
>>>>>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 420u IPv4 1020083846
>>>>>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 421u IPv4 1020083849
>>>>>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 422u IPv4 1020083865
>>>>>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 423u IPv4 1020083862
>>>>>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 424u IPv4 1020083877
>>>>>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 425u IPv4 1020083880
>>>>>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 426u IPv4 1020083888
>>>>>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 427u IPv4 1020083896
>>>>>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 433u IPv4 1020083914
>>>>>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 434u IPv4 1020083916
>>>>>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 435u IPv4 1020083926
>>>>>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 436u IPv4 1020083929
>>>>>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 437u IPv4 1020084012
>>>>>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 438u IPv4 1020083947
>>>>>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 439u IPv4 1020083975
>>>>>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 443u IPv4 1020083950
>>>>>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 444u IPv4 1020083959
>>>>>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 445u IPv4 1020083962
>>>>>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 446u IPv4 1020083977
>>>>>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 447u IPv4 1020083984
>>>>>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 448u IPv4 1020088663
>>>>>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 449u IPv4 1020083991
>>>>>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 450u IPv4 1020088678
>>>>>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 451u IPv4 1020088692
>>>>>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 452u IPv4 1020088694
>>>>>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 453u IPv4 1020092163
>>>>>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 5888 dwh 454u IPv4 1020092165
>>>>>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>>
>>>>>>>> Andy Kent wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have included the output from lsof, Unfortunately though I only
>>>>>>>>> restarted the server a few mins ago and so it doesn't really
>>>>>>>>> illustrate
>>>>>>>>> the
>>>>>>>>> problem. I'll try to post back with the same command tomorrow so
>>>>>>>>> that
>>>>>>>>> we
>>>>>>>>> can
>>>>>>>>> diff between the two.
>>>>>>>>>
>>>>>>>>> Thanks, Andy.
>>>>>>>>>
>>>>>>>>> java 16908 hadoop cwd DIR 0,17
>>>>>>>>> 240
>>>>>>>>> 7037865 /var/run/hadoop-0.20
>>>>>>>>> java 16908 hadoop rtd DIR 8,1
>>>>>>>>> 4096
>>>>>>>>> 2 /
>>>>>>>>> java 16908 hadoop txt REG 8,1
>>>>>>>>> 50810
>>>>>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 127296
>>>>>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 69472
>>>>>>>>> 3735429
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 44257
>>>>>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 80760
>>>>>>>>> 1164352 /lib/libresolv-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 22856
>>>>>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1145980
>>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 38105
>>>>>>>>> 3555614
>>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 111901
>>>>>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 392124
>>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 229928
>>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 724225
>>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 391834
>>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 169666
>>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 114342
>>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 121070
>>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 45946
>>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 87325
>>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 179508
>>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 20202
>>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 95488
>>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 390114
>>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 500917
>>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 31931
>>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 19349
>>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1798010
>>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 11412
>>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 14099
>>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 12641
>>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 7092
>>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 2446767
>>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1154170
>>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 176174
>>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1923209
>>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 26202
>>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 38015
>>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 261809
>>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 575389
>>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 46725
>>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 258337
>>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 43035
>>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 105209
>>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 134910
>>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1024681
>>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 15010
>>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 8601
>>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 15345
>>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 132368
>>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 65261
>>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 540852
>>>>>>>>> 3721351
>>>>>>>>> /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 391834
>>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 73374
>>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 114342
>>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 11428
>>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 198945
>>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 121070
>>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 163121
>>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 516429
>>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 321806
>>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 76698
>>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 405086
>>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 706710
>>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 706710
>>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 37210
>>>>>>>>> 6038622
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 18684
>>>>>>>>> 3721361
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 37157
>>>>>>>>> 3721359
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 3566844
>>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 180792
>>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 26202
>>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 38015
>>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 279781
>>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 2771746
>>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 12588713
>>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 25700
>>>>>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 254076
>>>>>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 51669471
>>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 92117
>>>>>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 47528
>>>>>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 43480
>>>>>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 35632
>>>>>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 55850
>>>>>>>>> 3555581
>>>>>>>>>
>>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 93080
>>>>>>>>> 1164343 /lib/libnsl-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 234713
>>>>>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 66580
>>>>>>>>> 3555574
>>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 35784
>>>>>>>>> 1164353 /lib/librt-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 526560
>>>>>>>>> 1164341 /lib/libm-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 10987257
>>>>>>>>> 3555599
>>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 1436976
>>>>>>>>> 1164337 /lib/libc-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 14624
>>>>>>>>> 1164340 /lib/libdl-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 130224
>>>>>>>>> 1164351 /lib/libpthread-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 127480
>>>>>>>>> 1164334 /lib/ld-2.7.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 618950
>>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 112341
>>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 46725
>>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 32768
>>>>>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 48335
>>>>>>>>> 3555565
>>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>>>>>> java 16908 hadoop mem REG 8,1
>>>>>>>>> 41123
>>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>>> java 16908 hadoop 0u CHR 136,2
>>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>>> java 16908 hadoop 1u CHR 136,2
>>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>>> java 16908 hadoop 2u CHR 136,2
>>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>>> java 16908 hadoop 3r REG 8,1
>>>>>>>>> 51669471
>>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>>> java 16908 hadoop 4r REG 8,1
>>>>>>>>> 12588713
>>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>>> java 16908 hadoop 5r REG 8,1
>>>>>>>>> 2771746
>>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>>> java 16908 hadoop 6r REG 8,1
>>>>>>>>> 41123
>>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>>> java 16908 hadoop 7r REG 8,1
>>>>>>>>> 46725
>>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>>> java 16908 hadoop 8r REG 8,1
>>>>>>>>> 112341
>>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>>> java 16908 hadoop 9r REG 8,1
>>>>>>>>> 279781
>>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>>> java 16908 hadoop 10r REG 8,1
>>>>>>>>> 38015
>>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>>> java 16908 hadoop 11r REG 8,1
>>>>>>>>> 26202
>>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>>> java 16908 hadoop 12r REG 8,1
>>>>>>>>> 180792
>>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>>> java 16908 hadoop 13r REG 8,1
>>>>>>>>> 3566844
>>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>>> java 16908 hadoop 14r REG 8,1
>>>>>>>>> 37157
>>>>>>>>> 3721359
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>>> java 16908 hadoop 15r REG 8,1
>>>>>>>>> 18684
>>>>>>>>> 3721361
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>>> java 16908 hadoop 16r REG 8,1
>>>>>>>>> 37210
>>>>>>>>> 6038622
>>>>>>>>> /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>>> java 16908 hadoop 17r REG 8,1
>>>>>>>>> 706710
>>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>>> java 16908 hadoop 18r REG 8,1
>>>>>>>>> 706710
>>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>>> java 16908 hadoop 19r REG 8,1
>>>>>>>>> 405086
>>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>>> java 16908 hadoop 20r REG 8,1
>>>>>>>>> 76698
>>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>>> java 16908 hadoop 21r REG 8,1
>>>>>>>>> 321806
>>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>>> java 16908 hadoop 22r REG 8,1
>>>>>>>>> 516429
>>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>>> java 16908 hadoop 23r REG 8,1
>>>>>>>>> 163121
>>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>>> java 16908 hadoop 24r REG 8,1
>>>>>>>>> 121070
>>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>>> java 16908 hadoop 25r REG 8,1
>>>>>>>>> 198945
>>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>>> java 16908 hadoop 26r REG 8,1
>>>>>>>>> 11428
>>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>>> java 16908 hadoop 27r REG 8,1
>>>>>>>>> 114342
>>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>>> java 16908 hadoop 28r REG 8,1
>>>>>>>>> 73374
>>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>>> java 16908 hadoop 29r REG 8,1
>>>>>>>>> 391834
>>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>>> java 16908 hadoop 30r REG 8,1
>>>>>>>>> 540852
>>>>>>>>> 3721351
>>>>>>>>> /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>>> java 16908 hadoop 31r REG 8,1
>>>>>>>>> 65261
>>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>>> java 16908 hadoop 32r REG 8,1
>>>>>>>>> 132368
>>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>>> java 16908 hadoop 33r REG 8,1
>>>>>>>>> 15345
>>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>>> java 16908 hadoop 34r REG 8,1
>>>>>>>>> 8601
>>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>>> java 16908 hadoop 35r REG 8,1
>>>>>>>>> 15010
>>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>>> java 16908 hadoop 36r REG 8,1
>>>>>>>>> 1024681
>>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>>> java 16908 hadoop 37r REG 8,1
>>>>>>>>> 134910
>>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>>> java 16908 hadoop 38r REG 8,1
>>>>>>>>> 105209
>>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>>> java 16908 hadoop 39r REG 8,1
>>>>>>>>> 43035
>>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>>> java 16908 hadoop 40r REG 8,1
>>>>>>>>> 258337
>>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>>> java 16908 hadoop 41r REG 8,1
>>>>>>>>> 46725
>>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>>> java 16908 hadoop 42r REG 8,1
>>>>>>>>> 575389
>>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>>> java 16908 hadoop 43r REG 8,1
>>>>>>>>> 261809
>>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>>> java 16908 hadoop 44r REG 8,1
>>>>>>>>> 38015
>>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>>> java 16908 hadoop 45r REG 8,1
>>>>>>>>> 26202
>>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>>> java 16908 hadoop 46r REG 8,1
>>>>>>>>> 1923209
>>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>>> java 16908 hadoop 47r REG 8,1
>>>>>>>>> 176174
>>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>>> java 16908 hadoop 48r REG 8,1
>>>>>>>>> 1154170
>>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>>> java 16908 hadoop 49r REG 8,1
>>>>>>>>> 2446767
>>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>>> java 16908 hadoop 50r REG 8,1
>>>>>>>>> 7092
>>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>>> java 16908 hadoop 51r REG 8,1
>>>>>>>>> 12641
>>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>>> java 16908 hadoop 52r REG 8,1
>>>>>>>>> 14099
>>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>>> java 16908 hadoop 53r REG 8,1
>>>>>>>>> 11412
>>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>>> java 16908 hadoop 54r REG 8,1
>>>>>>>>> 1798010
>>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>>> java 16908 hadoop 55r REG 8,1
>>>>>>>>> 19349
>>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>>> java 16908 hadoop 56r REG 8,1
>>>>>>>>> 31931
>>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>>> java 16908 hadoop 57r REG 8,1
>>>>>>>>> 500917
>>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>>> java 16908 hadoop 58r REG 8,1
>>>>>>>>> 390114
>>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>>> java 16908 hadoop 59r REG 8,1
>>>>>>>>> 95488
>>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>>> java 16908 hadoop 60r REG 8,1
>>>>>>>>> 20202
>>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>>> java 16908 hadoop 61r REG 8,1
>>>>>>>>> 179508
>>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>>> java 16908 hadoop 62r REG 8,1
>>>>>>>>> 87325
>>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>>> java 16908 hadoop 63r REG 8,1
>>>>>>>>> 45946
>>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>>> java 16908 hadoop 64r REG 8,1
>>>>>>>>> 121070
>>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>>> java 16908 hadoop 65r REG 8,1
>>>>>>>>> 114342
>>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>>> java 16908 hadoop 66r REG 8,1
>>>>>>>>> 169666
>>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>>> java 16908 hadoop 67r REG 8,1
>>>>>>>>> 391834
>>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>>> java 16908 hadoop 68r REG 8,1
>>>>>>>>> 724225
>>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>>> java 16908 hadoop 69r REG 8,1
>>>>>>>>> 229928
>>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>>> java 16908 hadoop 70r REG 8,1
>>>>>>>>> 392124
>>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>>> java 16908 hadoop 71r CHR 1,8
>>>>>>>>> 1818 /dev/random
>>>>>>>>> java 16908 hadoop 72r CHR 1,9
>>>>>>>>> 1824 /dev/urandom
>>>>>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>>>>>> TCP *:webmin (LISTEN)
>>>>>>>>> java 16908 hadoop 74u sock 0,4
>>>>>>>>> 153142739 can't identify protocol
>>>>>>>>> java 16908 hadoop 76w REG 8,1
>>>>>>>>> 6867
>>>>>>>>> 5043336
>>>>>>>>> /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 78r REG 8,1
>>>>>>>>> 1145980
>>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 81w REG 8,1
>>>>>>>>> 15491
>>>>>>>>> 5042961
>>>>>>>>> /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 83r REG 8,1
>>>>>>>>> 618950
>>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>>>>>> 153143422 socket
>>>>>>>>> java 16908 hadoop 86w REG 8,1
>>>>>>>>> 62510
>>>>>>>>> 5043340
>>>>>>>>> /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>>>>>> java 16908 hadoop 87w REG 8,1
>>>>>>>>> 960
>>>>>>>>> 5042962
>>>>>>>>> /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 113w REG 8,1
>>>>>>>>> 11562
>>>>>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>>>>>> 154318314 pipe
>>>>>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (ESTABLISHED)
>>>>>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>>>>>> 154318314 pipe
>>>>>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 249w REG 8,1
>>>>>>>>> 15230
>>>>>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 253u 0000 0,6
>>>>>>>>> 0
>>>>>>>>> 493 /anon_inode:[eventpoll]
>>>>>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>>> (CLOSE_WAIT)
>>>>>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345"
>>>>>>>>>> where
>>>>>>>>>> 12345 is the process id of the hive server?
>>>>>>>>>> We should be able to see the names of the files that are open.
>>>>>>>>>>
>>>>>>>>>> Zheng
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent
>>>>>>>>>> <An...@forward.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nope, no luck so far.
>>>>>>>>>>>
>>>>>>>>>>> We have upped the number of file descriptors and are having to
>>>>>>>>>>> restart
>>>>>>>>>>> hive every week or so :(
>>>>>>>>>>>
>>>>>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Did this help? I'm running into a similar problem. slowly
>>>>>>>>>>>> leaking
>>>>>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>>>>>
>>>>>>>>>>>> Andy Kent wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can give try and give it a go. I'm not convinced though as we
>>>>>>>>>>>>> are
>>>>>>>>>>>>> working with CSV files and don't touch sequence files at all at
>>>>>>>>>>>>> the
>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133
>>>>>>>>>>>>> and
>>>>>>>>>>>>> Hive 0.40
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Actually, we had an issue with this, it was a bug in
>>>>>>>>>>>>>> SequenceFile
>>>>>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>>>>>> filehandle
>>>>>>>>>>>>>> open and never close it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you
>>>>>>>>>>>>>> apply
>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could
>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>> combination.. maybe if you're doing queries and then closing
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> thrift
>>>>>>>>>>>>>> connection before reading all results, Hive doesn't know what
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> leaves the connection open? Once the west coast folks wake
>>>>>>>>>>>>>> up,
>>>>>>>>>>>>>> they might
>>>>>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>>>>>> <An...@forward.co.uk>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's
>>>>>>>>>>>>>>> connecting
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?)
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's what I was suspecting. I have triple checked and our
>>>>>>>>>>>>>> Ruby
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll try running some different queries and see if I can suss
>>>>>>>>>>>>>> out
>>>>>>>>>>>>>> some examples of which ones are leaky. Is this something that
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> should post
>>>>>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>>>>>> haven't noticed
>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Yours,
>>>>>>>>>> Zheng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Yours,
>>>>>>> Zheng
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Yours,
>>>>> Zheng
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Sent from my mobile device
>>>
>>> Yours,
>>> Zheng
>>>
>>>
>>
>>
>>
>
>
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
I've tried to look into it a bit more and it seems to happen on "load
data inpath"
LOAD DATA INPATH '/user/hive/warehouse/chatsessions_2010-02-01.csv' INTO
TABLE chatsessions_load;
And not on things like
select count(1) from chatsessions_load;
Bennie Schut wrote:
> That's 133k of data so perhaps this is enough?:
>
> 627: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 883: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 934: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 944: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> After a couple more queries:
>
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 991: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> some more stuff
> cat jmap.txt | grep hadoop.fs
> 535: 8 192
> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
> 539: 8 192 org.apache.hadoop.fs.permission.FsAction
> 611: 8 128 org.apache.hadoop.fs.Path
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 721: 4 96
> org.apache.hadoop.fs.permission.FsPermission$2
> 760: 2 96
> [Lorg.apache.hadoop.fs.permission.FsAction;
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1213: 1 24
> org.apache.hadoop.fs.permission.FsPermission
> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
> 1413: 1 16
> org.apache.hadoop.fs.ChecksumFileSystem$1
> 1523: 1 16
> org.apache.hadoop.fs.permission.FsPermission$1
> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
>
>
> Currently:
> lsof | grep "50010 (ESTABLISHED)" | wc -l
> 453
>
> Zheng Shao wrote:
>
>> Thanks for the quick reply. It seems the number of threads are normal.
>>
>> Can you do "jmap -histo:live" as well to find out the number of
>> DFSClient, FileSystem, etc?
>>
>>
>> On 2/16/10, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>>
>>> jstack on the Hive process:
>>>
>>> 2010-02-16
>>> 09:30:47
>>>
>>>
>>>
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>> mode):
>>>
>>>
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>>> condition [0x000000004238d000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
>>> condition [0x0000000042690000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
>>> on condition [0x0000000042791000]
>>> java.lang.Thread.State: TIMED_WAITING
>>> (sleeping)
>>> at java.lang.Thread.$$YJP$$sleep(Native
>>> Method)
>>> at
>>> java.lang.Thread.sleep(Thread.java)
>>>
>>>
>>> at
>>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
>>> condition [0x000000004258f000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
>>> condition [0x000000004248e000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
>>> condition [0x0000000040c7e000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>
>>>
>>> at
>>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
>>> nid=0x179d runnable [0x000000004228c000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>> at java.net.SocketInputStream.socketRead0(Native
>>> Method)
>>> at
>>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>
>>>
>>> at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>
>>>
>>> - locked <0x00007fbf10922580> (a
>>> java.io.BufferedInputStream)
>>> at
>>> java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>
>>>
>>> at
>>> java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>
>>>
>>> at
>>> com.yourkit.runtime.Packet.readHeader(Packet.java:40)
>>>
>>>
>>> at
>>> com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
>>>
>>>
>>> at
>>> com.yourkit.runtime.Core$4.run(Core.java:525)
>>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
>>> waiting on condition [0x0000000040d7f000]
>>> java.lang.Thread.State: TIMED_WAITING
>>> (sleeping)
>>> at java.lang.Thread.$$YJP$$sleep(Native
>>> Method)
>>> at
>>> java.lang.Thread.sleep(Thread.java)
>>>
>>>
>>> at
>>> com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
>>>
>>>
>>>
>>> "Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
>>> nid=0x1795 runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
>>> runnable [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>> "Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
>>> Object.wait() [0x000000004119d000]
>>> java.lang.Thread.State: WAITING (on object
>>> monitor)
>>> at java.lang.Object.wait(Native
>>> Method)
>>> - waiting on <0x00007fbea52c6790> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>>
>>>
>>> - locked <0x00007fbea52c6790> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>>
>>>
>>> at
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>
>>>
>>>
>>> "Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
>>> Object.wait() [0x000000004208a000]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> - waiting on <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>> at java.lang.Object.wait(Object.java:485)
>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> - locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>>
>>> "main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
>>> [0x00000000414e3000]
>>> java.lang.Thread.State: RUNNABLE
>>> at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>>> - locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>>> at
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
>>> at
>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>> at
>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
>>> at
>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>> "VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
>>>
>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
>>> nid=0x1782 runnable
>>>
>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
>>> nid=0x1783 runnable
>>>
>>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
>>> nid=0x1784 runnable
>>>
>>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
>>> nid=0x1785 runnable
>>>
>>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
>>> nid=0x1786 runnable
>>>
>>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
>>> nid=0x1787 runnable
>>>
>>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
>>> nid=0x1788 runnable
>>>
>>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
>>> nid=0x1789 runnable
>>>
>>> "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
>>> nid=0x178a runnable
>>>
>>> "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
>>> nid=0x178b runnable
>>>
>>> "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
>>> nid=0x178c runnable
>>>
>>> "GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
>>> nid=0x178d runnable
>>>
>>> "GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
>>> nid=0x178e runnable
>>>
>>> "VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
>>> waiting on condition
>>>
>>> JNI global references: 375194
>>>
>>>
>>>
>>> Zheng Shao wrote:
>>>
>>>
>>>> It seems that Hive server is leaking hdfs handles.
>>>> Not sure if hadoop has a command to show open hdfs handles.
>>>>
>>>> Can you also do a jstack of the Hive Server process?
>>>> First, "ps -elf | grep java" to find the hive server process id, and
>>>> the jdk directory
>>>> Second, "<jdk directory>/bin/jstack <pid>".
>>>>
>>>> This will tell us if there are clients of the Hive Server that didn't
>>>> close their connections yet.
>>>>
>>>> Zheng
>>>>
>>>> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>>>>
>>>>
>>>>
>>>>> That's the datanode port. So Hdfs. Any trick to see what is opened on
>>>>> hdfs?
>>>>> like an lsof for hdfs :)
>>>>>
>>>>> Zheng Shao wrote:
>>>>>
>>>>>
>>>>>
>>>>>> What's running at victoria.ebuddy.com:50010 ?
>>>>>>
>>>>>> Zhemg
>>>>>>
>>>>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I think it's lines like this:
>>>>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>>
>>>>>>> We have the same.
>>>>>>> Currently no queries running and many of those connections open. After
>>>>>>> running some queries a few more are added to the list.
>>>>>>>
>>>>>>> java 5888 dwh 389u IPv4 1020083623
>>>>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 390u IPv4 1020083626
>>>>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 391u IPv4 1020083637
>>>>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 392u IPv4 1020083641
>>>>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 393u IPv4 1020083668
>>>>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 394u IPv4 1020083675
>>>>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 395u IPv4 1020083678
>>>>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 396u IPv4 1020083690
>>>>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 397u IPv4 1020083693
>>>>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 398u IPv4 1020083703
>>>>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 399u IPv4 1020083706
>>>>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 400u IPv4 1020083713
>>>>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 401u IPv4 1020083716
>>>>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 402u IPv4 1020083728
>>>>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 403u IPv4 1020083731
>>>>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 404u IPv4 1020083740
>>>>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 405u IPv4 1020083743
>>>>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 406u IPv4 1020083753
>>>>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 407u IPv4 1020083756
>>>>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 408u IPv4 1020083769
>>>>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 409u IPv4 1020083772
>>>>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 410u IPv4 1020083781
>>>>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 411u IPv4 1020083784
>>>>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 412u IPv4 1020083792
>>>>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 413u IPv4 1020083795
>>>>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 414u IPv4 1020083808
>>>>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 415u IPv4 1020083811
>>>>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 416u IPv4 1020083820
>>>>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 417u IPv4 1020083823
>>>>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 418u IPv4 1020083831
>>>>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 419u IPv4 1020083834
>>>>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 420u IPv4 1020083846
>>>>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 421u IPv4 1020083849
>>>>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 422u IPv4 1020083865
>>>>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 423u IPv4 1020083862
>>>>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 424u IPv4 1020083877
>>>>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 425u IPv4 1020083880
>>>>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 426u IPv4 1020083888
>>>>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 427u IPv4 1020083896
>>>>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 433u IPv4 1020083914
>>>>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 434u IPv4 1020083916
>>>>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 435u IPv4 1020083926
>>>>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 436u IPv4 1020083929
>>>>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 437u IPv4 1020084012
>>>>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 438u IPv4 1020083947
>>>>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 439u IPv4 1020083975
>>>>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 443u IPv4 1020083950
>>>>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 444u IPv4 1020083959
>>>>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 445u IPv4 1020083962
>>>>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 446u IPv4 1020083977
>>>>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 447u IPv4 1020083984
>>>>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 448u IPv4 1020088663
>>>>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 449u IPv4 1020083991
>>>>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 450u IPv4 1020088678
>>>>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 451u IPv4 1020088692
>>>>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 452u IPv4 1020088694
>>>>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 453u IPv4 1020092163
>>>>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>> java 5888 dwh 454u IPv4 1020092165
>>>>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>>
>>>>>>> Andy Kent wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I have included the output from lsof, Unfortunately though I only
>>>>>>>> restarted the server a few mins ago and so it doesn't really
>>>>>>>> illustrate
>>>>>>>> the
>>>>>>>> problem. I'll try to post back with the same command tomorrow so that
>>>>>>>> we
>>>>>>>> can
>>>>>>>> diff between the two.
>>>>>>>>
>>>>>>>> Thanks, Andy.
>>>>>>>>
>>>>>>>> java 16908 hadoop cwd DIR 0,17 240
>>>>>>>> 7037865 /var/run/hadoop-0.20
>>>>>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>>>>>> 2 /
>>>>>>>> java 16908 hadoop txt REG 8,1 50810
>>>>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>>>>> java 16908 hadoop mem REG 8,1 127296
>>>>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>>>>> java 16908 hadoop mem REG 8,1 69472
>>>>>>>> 3735429
>>>>>>>>
>>>>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>>>>> java 16908 hadoop mem REG 8,1 44257
>>>>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>>>>> java 16908 hadoop mem REG 8,1 80760
>>>>>>>> 1164352 /lib/libresolv-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 22856
>>>>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 1145980
>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38105
>>>>>>>> 3555614
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>>>>> java 16908 hadoop mem REG 8,1 111901
>>>>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>>>>> java 16908 hadoop mem REG 8,1 392124
>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 229928
>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 724225
>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 169666
>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 45946
>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 87325
>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 179508
>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 20202
>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 95488
>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 390114
>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 500917
>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 31931
>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 19349
>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1798010
>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 11412
>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 14099
>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 12641
>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 7092
>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 2446767
>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1154170
>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 176174
>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1923209
>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 261809
>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 575389
>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 258337
>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 43035
>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 105209
>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 134910
>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 1024681
>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 15010
>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 8601
>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 15345
>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 132368
>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 65261
>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 540852
>>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 73374
>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 11428
>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 198945
>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 163121
>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 516429
>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 321806
>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 76698
>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 405086
>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 37210
>>>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 18684
>>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 37157
>>>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 3566844
>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 180792
>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 279781
>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 2771746
>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 12588713
>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 25700
>>>>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>>>>> java 16908 hadoop mem REG 8,1 254076
>>>>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>>>>> java 16908 hadoop mem REG 8,1 51669471
>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 92117
>>>>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>>>>> java 16908 hadoop mem REG 8,1 47528
>>>>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 43480
>>>>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 35632
>>>>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 55850
>>>>>>>> 3555581
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>>>>> java 16908 hadoop mem REG 8,1 93080
>>>>>>>> 1164343 /lib/libnsl-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 234713
>>>>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>>>>> java 16908 hadoop mem REG 8,1 66580
>>>>>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>>>>> java 16908 hadoop mem REG 8,1 35784
>>>>>>>> 1164353 /lib/librt-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 526560
>>>>>>>> 1164341 /lib/libm-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 10987257
>>>>>>>> 3555599
>>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>>>>> java 16908 hadoop mem REG 8,1 1436976
>>>>>>>> 1164337 /lib/libc-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 14624
>>>>>>>> 1164340 /lib/libdl-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 130224
>>>>>>>> 1164351 /lib/libpthread-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 127480
>>>>>>>> 1164334 /lib/ld-2.7.so
>>>>>>>> java 16908 hadoop mem REG 8,1 618950
>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 112341
>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop mem REG 8,1 32768
>>>>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>>>>> java 16908 hadoop mem REG 8,1 48335
>>>>>>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>>>>> java 16908 hadoop mem REG 8,1 41123
>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>> java 16908 hadoop 0u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 1u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 2u CHR 136,2
>>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>>> java 16908 hadoop 6r REG 8,1 41123
>>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>>> java 16908 hadoop 7r REG 8,1 46725
>>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop 8r REG 8,1 112341
>>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>>> java 16908 hadoop 9r REG 8,1 279781
>>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>>> java 16908 hadoop 10r REG 8,1 38015
>>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop 11r REG 8,1 26202
>>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop 12r REG 8,1 180792
>>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>>> java 16908 hadoop 14r REG 8,1 37157
>>>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>>> java 16908 hadoop 15r REG 8,1 18684
>>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>>> java 16908 hadoop 16r REG 8,1 37210
>>>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>>> java 16908 hadoop 17r REG 8,1 706710
>>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>>> java 16908 hadoop 18r REG 8,1 706710
>>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>>> java 16908 hadoop 19r REG 8,1 405086
>>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>>> java 16908 hadoop 20r REG 8,1 76698
>>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>>> java 16908 hadoop 21r REG 8,1 321806
>>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>>> java 16908 hadoop 22r REG 8,1 516429
>>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>>> java 16908 hadoop 23r REG 8,1 163121
>>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>>> java 16908 hadoop 24r REG 8,1 121070
>>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop 25r REG 8,1 198945
>>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>>> java 16908 hadoop 26r REG 8,1 11428
>>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>>> java 16908 hadoop 27r REG 8,1 114342
>>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>>> java 16908 hadoop 28r REG 8,1 73374
>>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>>> java 16908 hadoop 29r REG 8,1 391834
>>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop 30r REG 8,1 540852
>>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>>> java 16908 hadoop 31r REG 8,1 65261
>>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>>> java 16908 hadoop 32r REG 8,1 132368
>>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>>> java 16908 hadoop 33r REG 8,1 15345
>>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>>> java 16908 hadoop 34r REG 8,1 8601
>>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>>> java 16908 hadoop 35r REG 8,1 15010
>>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>>> java 16908 hadoop 37r REG 8,1 134910
>>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>>> java 16908 hadoop 38r REG 8,1 105209
>>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>>> java 16908 hadoop 39r REG 8,1 43035
>>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>>> java 16908 hadoop 40r REG 8,1 258337
>>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop 41r REG 8,1 46725
>>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>>> java 16908 hadoop 42r REG 8,1 575389
>>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>>> java 16908 hadoop 43r REG 8,1 261809
>>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>>> java 16908 hadoop 44r REG 8,1 38015
>>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>>> java 16908 hadoop 45r REG 8,1 26202
>>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>>> java 16908 hadoop 47r REG 8,1 176174
>>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>>> java 16908 hadoop 50r REG 8,1 7092
>>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>>> java 16908 hadoop 51r REG 8,1 12641
>>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>>> java 16908 hadoop 52r REG 8,1 14099
>>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>>> java 16908 hadoop 53r REG 8,1 11412
>>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>>> java 16908 hadoop 55r REG 8,1 19349
>>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>>> java 16908 hadoop 56r REG 8,1 31931
>>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>>> java 16908 hadoop 57r REG 8,1 500917
>>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>>> java 16908 hadoop 58r REG 8,1 390114
>>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>>> java 16908 hadoop 59r REG 8,1 95488
>>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>>> java 16908 hadoop 60r REG 8,1 20202
>>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>>> java 16908 hadoop 61r REG 8,1 179508
>>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>>> java 16908 hadoop 62r REG 8,1 87325
>>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>>> java 16908 hadoop 63r REG 8,1 45946
>>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>>> java 16908 hadoop 64r REG 8,1 121070
>>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>>> java 16908 hadoop 65r REG 8,1 114342
>>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>>> java 16908 hadoop 66r REG 8,1 169666
>>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>>> java 16908 hadoop 67r REG 8,1 391834
>>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>>> java 16908 hadoop 68r REG 8,1 724225
>>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>>> java 16908 hadoop 69r REG 8,1 229928
>>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>>> java 16908 hadoop 70r REG 8,1 392124
>>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>>> java 16908 hadoop 71r CHR 1,8
>>>>>>>> 1818 /dev/random
>>>>>>>> java 16908 hadoop 72r CHR 1,9
>>>>>>>> 1824 /dev/urandom
>>>>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>>>>> TCP *:webmin (LISTEN)
>>>>>>>> java 16908 hadoop 74u sock 0,4
>>>>>>>> 153142739 can't identify protocol
>>>>>>>> java 16908 hadoop 76w REG 8,1 6867
>>>>>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 81w REG 8,1 15491
>>>>>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 83r REG 8,1 618950
>>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>>>>> 153143422 socket
>>>>>>>> java 16908 hadoop 86w REG 8,1 62510
>>>>>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>>>>> java 16908 hadoop 87w REG 8,1 960
>>>>>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 113w REG 8,1 11562
>>>>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>>>>> 154318314 pipe
>>>>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (ESTABLISHED)
>>>>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>>>>> 154318314 pipe
>>>>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 249w REG 8,1 15230
>>>>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 253u 0000 0,6 0
>>>>>>>> 493 /anon_inode:[eventpoll]
>>>>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>>>>> TCP
>>>>>>>>
>>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>>> (CLOSE_WAIT)
>>>>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345"
>>>>>>>>> where
>>>>>>>>> 12345 is the process id of the hive server?
>>>>>>>>> We should be able to see the names of the files that are open.
>>>>>>>>>
>>>>>>>>> Zheng
>>>>>>>>>
>>>>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Nope, no luck so far.
>>>>>>>>>>
>>>>>>>>>> We have upped the number of file descriptors and are having to
>>>>>>>>>> restart
>>>>>>>>>> hive every week or so :(
>>>>>>>>>>
>>>>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>>>>
>>>>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>>>>
>>>>>>>>>>> Andy Kent wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I can give try and give it a go. I'm not convinced though as we
>>>>>>>>>>>> are
>>>>>>>>>>>> working with CSV files and don't touch sequence files at all at
>>>>>>>>>>>> the
>>>>>>>>>>>> moment.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>>>>>>> Hive 0.40
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>>>>> filehandle
>>>>>>>>>>>>> open and never close it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get
>>>>>>>>>>>>> some
>>>>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>>>>>> this to
>>>>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>>>>>>> combination.. maybe if you're doing queries and then closing
>>>>>>>>>>>>> your
>>>>>>>>>>>>> thrift
>>>>>>>>>>>>> connection before reading all results, Hive doesn't know what to
>>>>>>>>>>>>> do
>>>>>>>>>>>>> and
>>>>>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>>>>>> they might
>>>>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby
>>>>>>>>>>>>> code
>>>>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>>>>>> should post
>>>>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>>>>> haven't noticed
>>>>>>>>>>>>> this?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Yours,
>>>>>>>>> Zheng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> --
>>>>>> Yours,
>>>>>> Zheng
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>>
>>>>
>> --
>> Sent from my mobile device
>>
>> Yours,
>> Zheng
>>
>>
>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
That's 133k of data so perhaps this is enough?:
627: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
723: 1 80 org.apache.hadoop.hdfs.DFSClient
882: 1 48 org.apache.hadoop.fs.LocalFileSystem
883: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
934: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
944: 1 48
org.apache.hadoop.hdfs.DistributedFileSystem
1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
After a couple more queries:
671: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
772: 1 80 org.apache.hadoop.hdfs.DFSClient
930: 1 48 org.apache.hadoop.fs.LocalFileSystem
931: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
981: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
991: 1 48
org.apache.hadoop.hdfs.DistributedFileSystem
1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
some more stuff
cat jmap.txt | grep hadoop.fs
535: 8 192
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
539: 8 192 org.apache.hadoop.fs.permission.FsAction
611: 8 128 org.apache.hadoop.fs.Path
671: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
721: 4 96
org.apache.hadoop.fs.permission.FsPermission$2
760: 2 96
[Lorg.apache.hadoop.fs.permission.FsAction;
930: 1 48 org.apache.hadoop.fs.LocalFileSystem
931: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
981: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1213: 1 24
org.apache.hadoop.fs.permission.FsPermission
1347: 1 16 org.apache.hadoop.fs.FileSystem$1
1413: 1 16
org.apache.hadoop.fs.ChecksumFileSystem$1
1523: 1 16
org.apache.hadoop.fs.permission.FsPermission$1
1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
Currently:
lsof | grep "50010 (ESTABLISHED)" | wc -l
453
Zheng Shao wrote:
> Thanks for the quick reply. It seems the number of threads are normal.
>
> Can you do "jmap -histo:live" as well to find out the number of
> DFSClient, FileSystem, etc?
>
>
> On 2/16/10, Bennie Schut <bs...@ebuddy.com> wrote:
>
>> jstack on the Hive process:
>>
>> 2010-02-16
>> 09:30:47
>>
>>
>>
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>> mode):
>>
>>
>>
>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>> condition [0x000000004238d000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
>> condition [0x0000000042690000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
>> on condition [0x0000000042791000]
>> java.lang.Thread.State: TIMED_WAITING
>> (sleeping)
>> at java.lang.Thread.$$YJP$$sleep(Native
>> Method)
>> at
>> java.lang.Thread.sleep(Thread.java)
>>
>>
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
>> condition [0x000000004258f000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
>> condition [0x000000004248e000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
>> condition [0x0000000040c7e000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
>> nid=0x179d runnable [0x000000004228c000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>> at java.net.SocketInputStream.socketRead0(Native
>> Method)
>> at
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>
>>
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>
>>
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>
>>
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>
>>
>> - locked <0x00007fbf10922580> (a
>> java.io.BufferedInputStream)
>> at
>> java.io.DataInputStream.readFully(DataInputStream.java:178)
>>
>>
>> at
>> java.io.DataInputStream.readLong(DataInputStream.java:399)
>>
>>
>> at
>> com.yourkit.runtime.Packet.readHeader(Packet.java:40)
>>
>>
>> at
>> com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
>>
>>
>> at
>> com.yourkit.runtime.Core$4.run(Core.java:525)
>>
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
>> waiting on condition [0x0000000040d7f000]
>> java.lang.Thread.State: TIMED_WAITING
>> (sleeping)
>> at java.lang.Thread.$$YJP$$sleep(Native
>> Method)
>> at
>> java.lang.Thread.sleep(Thread.java)
>>
>>
>> at
>> com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
>>
>>
>>
>> "Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
>> runnable [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>> "CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
>> nid=0x1795 runnable [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
>> runnable [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
>> runnable [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
>> Object.wait() [0x000000004119d000]
>> java.lang.Thread.State: WAITING (on object
>> monitor)
>> at java.lang.Object.wait(Native
>> Method)
>> - waiting on <0x00007fbea52c6790> (a
>> java.lang.ref.ReferenceQueue$Lock)
>> at
>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>
>>
>> - locked <0x00007fbea52c6790> (a
>> java.lang.ref.ReferenceQueue$Lock)
>> at
>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>
>>
>> at
>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>
>>
>>
>> "Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
>> Object.wait() [0x000000004208a000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>> at java.lang.Object.wait(Object.java:485)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> - locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>>
>> "main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
>> [0x00000000414e3000]
>> java.lang.Thread.State: RUNNABLE
>> at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
>> at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>> - locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>> at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
>> at
>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>> at
>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
>> at
>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> "VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
>>
>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
>> nid=0x1782 runnable
>>
>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
>> nid=0x1783 runnable
>>
>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
>> nid=0x1784 runnable
>>
>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
>> nid=0x1785 runnable
>>
>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
>> nid=0x1786 runnable
>>
>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
>> nid=0x1787 runnable
>>
>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
>> nid=0x1788 runnable
>>
>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
>> nid=0x1789 runnable
>>
>> "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
>> nid=0x178a runnable
>>
>> "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
>> nid=0x178b runnable
>>
>> "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
>> nid=0x178c runnable
>>
>> "GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
>> nid=0x178d runnable
>>
>> "GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
>> nid=0x178e runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
>> waiting on condition
>>
>> JNI global references: 375194
>>
>>
>>
>> Zheng Shao wrote:
>>
>>> It seems that Hive server is leaking hdfs handles.
>>> Not sure if hadoop has a command to show open hdfs handles.
>>>
>>> Can you also do a jstack of the Hive Server process?
>>> First, "ps -elf | grep java" to find the hive server process id, and
>>> the jdk directory
>>> Second, "<jdk directory>/bin/jstack <pid>".
>>>
>>> This will tell us if there are clients of the Hive Server that didn't
>>> close their connections yet.
>>>
>>> Zheng
>>>
>>> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>>
>>>> That's the datanode port. So Hdfs. Any trick to see what is opened on
>>>> hdfs?
>>>> like an lsof for hdfs :)
>>>>
>>>> Zheng Shao wrote:
>>>>
>>>>
>>>>> What's running at victoria.ebuddy.com:50010 ?
>>>>>
>>>>> Zhemg
>>>>>
>>>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> I think it's lines like this:
>>>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>>
>>>>>> We have the same.
>>>>>> Currently no queries running and many of those connections open. After
>>>>>> running some queries a few more are added to the list.
>>>>>>
>>>>>> java 5888 dwh 389u IPv4 1020083623
>>>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 390u IPv4 1020083626
>>>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 391u IPv4 1020083637
>>>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 392u IPv4 1020083641
>>>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 393u IPv4 1020083668
>>>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 394u IPv4 1020083675
>>>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 395u IPv4 1020083678
>>>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 396u IPv4 1020083690
>>>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 397u IPv4 1020083693
>>>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 398u IPv4 1020083703
>>>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 399u IPv4 1020083706
>>>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 400u IPv4 1020083713
>>>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 401u IPv4 1020083716
>>>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 402u IPv4 1020083728
>>>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 403u IPv4 1020083731
>>>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 404u IPv4 1020083740
>>>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 405u IPv4 1020083743
>>>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 406u IPv4 1020083753
>>>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 407u IPv4 1020083756
>>>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 408u IPv4 1020083769
>>>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 409u IPv4 1020083772
>>>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 410u IPv4 1020083781
>>>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 411u IPv4 1020083784
>>>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 412u IPv4 1020083792
>>>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 413u IPv4 1020083795
>>>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 414u IPv4 1020083808
>>>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 415u IPv4 1020083811
>>>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 416u IPv4 1020083820
>>>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 417u IPv4 1020083823
>>>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 418u IPv4 1020083831
>>>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 419u IPv4 1020083834
>>>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 420u IPv4 1020083846
>>>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 421u IPv4 1020083849
>>>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 422u IPv4 1020083865
>>>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 423u IPv4 1020083862
>>>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 424u IPv4 1020083877
>>>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 425u IPv4 1020083880
>>>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 426u IPv4 1020083888
>>>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 427u IPv4 1020083896
>>>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 433u IPv4 1020083914
>>>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 434u IPv4 1020083916
>>>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 435u IPv4 1020083926
>>>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 436u IPv4 1020083929
>>>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 437u IPv4 1020084012
>>>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 438u IPv4 1020083947
>>>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 439u IPv4 1020083975
>>>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 443u IPv4 1020083950
>>>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 444u IPv4 1020083959
>>>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 445u IPv4 1020083962
>>>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 446u IPv4 1020083977
>>>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 447u IPv4 1020083984
>>>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 448u IPv4 1020088663
>>>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 449u IPv4 1020083991
>>>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 450u IPv4 1020088678
>>>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 451u IPv4 1020088692
>>>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 452u IPv4 1020088694
>>>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 453u IPv4 1020092163
>>>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>> java 5888 dwh 454u IPv4 1020092165
>>>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>>
>>>>>> Andy Kent wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I have included the output from lsof, Unfortunately though I only
>>>>>>> restarted the server a few mins ago and so it doesn't really
>>>>>>> illustrate
>>>>>>> the
>>>>>>> problem. I'll try to post back with the same command tomorrow so that
>>>>>>> we
>>>>>>> can
>>>>>>> diff between the two.
>>>>>>>
>>>>>>> Thanks, Andy.
>>>>>>>
>>>>>>> java 16908 hadoop cwd DIR 0,17 240
>>>>>>> 7037865 /var/run/hadoop-0.20
>>>>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>>>>> 2 /
>>>>>>> java 16908 hadoop txt REG 8,1 50810
>>>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>>>> java 16908 hadoop mem REG 8,1 127296
>>>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>>>> java 16908 hadoop mem REG 8,1 69472
>>>>>>> 3735429
>>>>>>>
>>>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>>>> java 16908 hadoop mem REG 8,1 44257
>>>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>>>> java 16908 hadoop mem REG 8,1 80760
>>>>>>> 1164352 /lib/libresolv-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 22856
>>>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 1145980
>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>> java 16908 hadoop mem REG 8,1 38105
>>>>>>> 3555614
>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>>>> java 16908 hadoop mem REG 8,1 111901
>>>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>>>> java 16908 hadoop mem REG 8,1 392124
>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>> java 16908 hadoop mem REG 8,1 229928
>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 724225
>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>> java 16908 hadoop mem REG 8,1 169666
>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 45946
>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>> java 16908 hadoop mem REG 8,1 87325
>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>> java 16908 hadoop mem REG 8,1 179508
>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>> java 16908 hadoop mem REG 8,1 20202
>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>> java 16908 hadoop mem REG 8,1 95488
>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>> java 16908 hadoop mem REG 8,1 390114
>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>> java 16908 hadoop mem REG 8,1 500917
>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>> java 16908 hadoop mem REG 8,1 31931
>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>> java 16908 hadoop mem REG 8,1 19349
>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>> java 16908 hadoop mem REG 8,1 1798010
>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>> java 16908 hadoop mem REG 8,1 11412
>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>> java 16908 hadoop mem REG 8,1 14099
>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>> java 16908 hadoop mem REG 8,1 12641
>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>> java 16908 hadoop mem REG 8,1 7092
>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>> java 16908 hadoop mem REG 8,1 2446767
>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>> java 16908 hadoop mem REG 8,1 1154170
>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>> java 16908 hadoop mem REG 8,1 176174
>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>> java 16908 hadoop mem REG 8,1 1923209
>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>> java 16908 hadoop mem REG 8,1 261809
>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>> java 16908 hadoop mem REG 8,1 575389
>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>> java 16908 hadoop mem REG 8,1 258337
>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>> java 16908 hadoop mem REG 8,1 43035
>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 105209
>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 134910
>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 1024681
>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 15010
>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>> java 16908 hadoop mem REG 8,1 8601
>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>> java 16908 hadoop mem REG 8,1 15345
>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>> java 16908 hadoop mem REG 8,1 132368
>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>> java 16908 hadoop mem REG 8,1 65261
>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>> java 16908 hadoop mem REG 8,1 540852
>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>> java 16908 hadoop mem REG 8,1 73374
>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>> java 16908 hadoop mem REG 8,1 11428
>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>> java 16908 hadoop mem REG 8,1 198945
>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 163121
>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>> java 16908 hadoop mem REG 8,1 516429
>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>> java 16908 hadoop mem REG 8,1 321806
>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 76698
>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>> java 16908 hadoop mem REG 8,1 405086
>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>> java 16908 hadoop mem REG 8,1 37210
>>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>> java 16908 hadoop mem REG 8,1 18684
>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>> java 16908 hadoop mem REG 8,1 37157
>>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>> java 16908 hadoop mem REG 8,1 3566844
>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 180792
>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>> java 16908 hadoop mem REG 8,1 279781
>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>> java 16908 hadoop mem REG 8,1 2771746
>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>> java 16908 hadoop mem REG 8,1 12588713
>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>> java 16908 hadoop mem REG 8,1 25700
>>>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>>>> java 16908 hadoop mem REG 8,1 254076
>>>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>>>> java 16908 hadoop mem REG 8,1 51669471
>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>> java 16908 hadoop mem REG 8,1 92117
>>>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>>>> java 16908 hadoop mem REG 8,1 47528
>>>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 43480
>>>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 35632
>>>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 55850
>>>>>>> 3555581
>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>>>> java 16908 hadoop mem REG 8,1 93080
>>>>>>> 1164343 /lib/libnsl-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 234713
>>>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>>>> java 16908 hadoop mem REG 8,1 66580
>>>>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>>>> java 16908 hadoop mem REG 8,1 35784
>>>>>>> 1164353 /lib/librt-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 526560
>>>>>>> 1164341 /lib/libm-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 10987257
>>>>>>> 3555599
>>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>>>> java 16908 hadoop mem REG 8,1 1436976
>>>>>>> 1164337 /lib/libc-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 14624
>>>>>>> 1164340 /lib/libdl-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 130224
>>>>>>> 1164351 /lib/libpthread-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 127480
>>>>>>> 1164334 /lib/ld-2.7.so
>>>>>>> java 16908 hadoop mem REG 8,1 618950
>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>> java 16908 hadoop mem REG 8,1 112341
>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>> java 16908 hadoop mem REG 8,1 32768
>>>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>>>> java 16908 hadoop mem REG 8,1 48335
>>>>>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>>>> java 16908 hadoop mem REG 8,1 41123
>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>> java 16908 hadoop 0u CHR 136,2
>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>> java 16908 hadoop 1u CHR 136,2
>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>> java 16908 hadoop 2u CHR 136,2
>>>>>>> 4 /dev/pts/2 (deleted)
>>>>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>>> java 16908 hadoop 6r REG 8,1 41123
>>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>>> java 16908 hadoop 7r REG 8,1 46725
>>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>>> java 16908 hadoop 8r REG 8,1 112341
>>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>>> java 16908 hadoop 9r REG 8,1 279781
>>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>>> java 16908 hadoop 10r REG 8,1 38015
>>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>>> java 16908 hadoop 11r REG 8,1 26202
>>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>>> java 16908 hadoop 12r REG 8,1 180792
>>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>>> java 16908 hadoop 14r REG 8,1 37157
>>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>>> java 16908 hadoop 15r REG 8,1 18684
>>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>>> java 16908 hadoop 16r REG 8,1 37210
>>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>>> java 16908 hadoop 17r REG 8,1 706710
>>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>>> java 16908 hadoop 18r REG 8,1 706710
>>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>>> java 16908 hadoop 19r REG 8,1 405086
>>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>>> java 16908 hadoop 20r REG 8,1 76698
>>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>>> java 16908 hadoop 21r REG 8,1 321806
>>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>>> java 16908 hadoop 22r REG 8,1 516429
>>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>>> java 16908 hadoop 23r REG 8,1 163121
>>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>>> java 16908 hadoop 24r REG 8,1 121070
>>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>>> java 16908 hadoop 25r REG 8,1 198945
>>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>>> java 16908 hadoop 26r REG 8,1 11428
>>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>>> java 16908 hadoop 27r REG 8,1 114342
>>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>>> java 16908 hadoop 28r REG 8,1 73374
>>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>>> java 16908 hadoop 29r REG 8,1 391834
>>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>>> java 16908 hadoop 30r REG 8,1 540852
>>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>>> java 16908 hadoop 31r REG 8,1 65261
>>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>>> java 16908 hadoop 32r REG 8,1 132368
>>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>>> java 16908 hadoop 33r REG 8,1 15345
>>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>>> java 16908 hadoop 34r REG 8,1 8601
>>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>>> java 16908 hadoop 35r REG 8,1 15010
>>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>>> java 16908 hadoop 37r REG 8,1 134910
>>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>>> java 16908 hadoop 38r REG 8,1 105209
>>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>>> java 16908 hadoop 39r REG 8,1 43035
>>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>>> java 16908 hadoop 40r REG 8,1 258337
>>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>>> java 16908 hadoop 41r REG 8,1 46725
>>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>>> java 16908 hadoop 42r REG 8,1 575389
>>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>>> java 16908 hadoop 43r REG 8,1 261809
>>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>>> java 16908 hadoop 44r REG 8,1 38015
>>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>>> java 16908 hadoop 45r REG 8,1 26202
>>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>>> java 16908 hadoop 47r REG 8,1 176174
>>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>>> java 16908 hadoop 50r REG 8,1 7092
>>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>>> java 16908 hadoop 51r REG 8,1 12641
>>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>>> java 16908 hadoop 52r REG 8,1 14099
>>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>>> java 16908 hadoop 53r REG 8,1 11412
>>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>>> java 16908 hadoop 55r REG 8,1 19349
>>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>>> java 16908 hadoop 56r REG 8,1 31931
>>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>>> java 16908 hadoop 57r REG 8,1 500917
>>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>>> java 16908 hadoop 58r REG 8,1 390114
>>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>>> java 16908 hadoop 59r REG 8,1 95488
>>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>>> java 16908 hadoop 60r REG 8,1 20202
>>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>>> java 16908 hadoop 61r REG 8,1 179508
>>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>>> java 16908 hadoop 62r REG 8,1 87325
>>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>>> java 16908 hadoop 63r REG 8,1 45946
>>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>>> java 16908 hadoop 64r REG 8,1 121070
>>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>>> java 16908 hadoop 65r REG 8,1 114342
>>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>>> java 16908 hadoop 66r REG 8,1 169666
>>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>>> java 16908 hadoop 67r REG 8,1 391834
>>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>>> java 16908 hadoop 68r REG 8,1 724225
>>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>>> java 16908 hadoop 69r REG 8,1 229928
>>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>>> java 16908 hadoop 70r REG 8,1 392124
>>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>>> java 16908 hadoop 71r CHR 1,8
>>>>>>> 1818 /dev/random
>>>>>>> java 16908 hadoop 72r CHR 1,9
>>>>>>> 1824 /dev/urandom
>>>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>>>> TCP *:webmin (LISTEN)
>>>>>>> java 16908 hadoop 74u sock 0,4
>>>>>>> 153142739 can't identify protocol
>>>>>>> java 16908 hadoop 76w REG 8,1 6867
>>>>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 81w REG 8,1 15491
>>>>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 83r REG 8,1 618950
>>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>>>> 153143422 socket
>>>>>>> java 16908 hadoop 86w REG 8,1 62510
>>>>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>>>> java 16908 hadoop 87w REG 8,1 960
>>>>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 113w REG 8,1 11562
>>>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>>>> 154318314 pipe
>>>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (ESTABLISHED)
>>>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>>>> 154318314 pipe
>>>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 249w REG 8,1 15230
>>>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 253u 0000 0,6 0
>>>>>>> 493 /anon_inode:[eventpoll]
>>>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>>>> TCP
>>>>>>>
>>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>>> (CLOSE_WAIT)
>>>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345"
>>>>>>>> where
>>>>>>>> 12345 is the process id of the hive server?
>>>>>>>> We should be able to see the names of the files that are open.
>>>>>>>>
>>>>>>>> Zheng
>>>>>>>>
>>>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Nope, no luck so far.
>>>>>>>>>
>>>>>>>>> We have upped the number of file descriptors and are having to
>>>>>>>>> restart
>>>>>>>>> hive every week or so :(
>>>>>>>>>
>>>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>>>
>>>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>>>
>>>>>>>>>> Andy Kent wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I can give try and give it a go. I'm not convinced though as we
>>>>>>>>>>> are
>>>>>>>>>>> working with CSV files and don't touch sequence files at all at
>>>>>>>>>>> the
>>>>>>>>>>> moment.
>>>>>>>>>>>
>>>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>>>>>> Hive 0.40
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>>>> filehandle
>>>>>>>>>>>> open and never close it.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get
>>>>>>>>>>>> some
>>>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>>>>> this to
>>>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>>>>>> combination.. maybe if you're doing queries and then closing
>>>>>>>>>>>> your
>>>>>>>>>>>> thrift
>>>>>>>>>>>> connection before reading all results, Hive doesn't know what to
>>>>>>>>>>>> do
>>>>>>>>>>>> and
>>>>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>>>>> they might
>>>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting
>>>>>>>>>>>>> to
>>>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and
>>>>>>>>>>>>> not
>>>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>
>>>>>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby
>>>>>>>>>>>> code
>>>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>>>>> should post
>>>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>>>> haven't noticed
>>>>>>>>>>>> this?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Yours,
>>>>>>>> Zheng
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> Yours,
>>>>> Zheng
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>>
>>
>
> --
> Sent from my mobile device
>
> Yours,
> Zheng
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
Thanks for the quick reply. It seems the number of threads are normal.
Can you do "jmap -histo:live" as well to find out the number of
DFSClient, FileSystem, etc?
On 2/16/10, Bennie Schut <bs...@ebuddy.com> wrote:
> jstack on the Hive process:
>
> 2010-02-16
> 09:30:47
>
>
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
> mode):
>
>
>
> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
>
>
> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
> condition [0x000000004238d000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
> condition [0x0000000042690000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
> on condition [0x0000000042791000]
> java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> at java.lang.Thread.$$YJP$$sleep(Native
> Method)
> at
> java.lang.Thread.sleep(Thread.java)
>
>
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
> condition [0x000000004258f000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
> condition [0x000000004248e000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
> condition [0x0000000040c7e000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
> nid=0x179d runnable [0x000000004228c000]
> java.lang.Thread.State:
> RUNNABLE
>
>
> at java.net.SocketInputStream.socketRead0(Native
> Method)
> at
> java.net.SocketInputStream.read(SocketInputStream.java:129)
>
>
> at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>
>
> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>
>
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>
>
> - locked <0x00007fbf10922580> (a
> java.io.BufferedInputStream)
> at
> java.io.DataInputStream.readFully(DataInputStream.java:178)
>
>
> at
> java.io.DataInputStream.readLong(DataInputStream.java:399)
>
>
> at
> com.yourkit.runtime.Packet.readHeader(Packet.java:40)
>
>
> at
> com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
>
>
> at
> com.yourkit.runtime.Core$4.run(Core.java:525)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
> waiting on condition [0x0000000040d7f000]
> java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> at java.lang.Thread.$$YJP$$sleep(Native
> Method)
> at
> java.lang.Thread.sleep(Thread.java)
>
>
> at
> com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
>
>
>
> "Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
> runnable [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
>
>
> "CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
>
>
> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
> nid=0x1795 runnable [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
>
>
> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
> runnable [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
> runnable [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
> Object.wait() [0x000000004119d000]
> java.lang.Thread.State: WAITING (on object
> monitor)
> at java.lang.Object.wait(Native
> Method)
> - waiting on <0x00007fbea52c6790> (a
> java.lang.ref.ReferenceQueue$Lock)
> at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>
>
> - locked <0x00007fbea52c6790> (a
> java.lang.ref.ReferenceQueue$Lock)
> at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>
>
> at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>
>
> "Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
> Object.wait() [0x000000004208a000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
> [0x00000000414e3000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
> at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
> - locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
> at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
> at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
> at
> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
> at
> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> "VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
> nid=0x1782 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
> nid=0x1783 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
> nid=0x1784 runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
> nid=0x1785 runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
> nid=0x1786 runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
> nid=0x1787 runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
> nid=0x1788 runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
> nid=0x1789 runnable
>
> "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
> nid=0x178a runnable
>
> "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
> nid=0x178b runnable
>
> "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
> nid=0x178c runnable
>
> "GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
> nid=0x178d runnable
>
> "GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
> nid=0x178e runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
> waiting on condition
>
> JNI global references: 375194
>
>
>
> Zheng Shao wrote:
>> It seems that Hive server is leaking hdfs handles.
>> Not sure if hadoop has a command to show open hdfs handles.
>>
>> Can you also do a jstack of the Hive Server process?
>> First, "ps -elf | grep java" to find the hive server process id, and
>> the jdk directory
>> Second, "<jdk directory>/bin/jstack <pid>".
>>
>> This will tell us if there are clients of the Hive Server that didn't
>> close their connections yet.
>>
>> Zheng
>>
>> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>>> That's the datanode port. So Hdfs. Any trick to see what is opened on
>>> hdfs?
>>> like an lsof for hdfs :)
>>>
>>> Zheng Shao wrote:
>>>
>>>> What's running at victoria.ebuddy.com:50010 ?
>>>>
>>>> Zhemg
>>>>
>>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com>
>>>> wrote:
>>>>
>>>>
>>>>> I think it's lines like this:
>>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>>
>>>>> We have the same.
>>>>> Currently no queries running and many of those connections open. After
>>>>> running some queries a few more are added to the list.
>>>>>
>>>>> java 5888 dwh 389u IPv4 1020083623
>>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 390u IPv4 1020083626
>>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 391u IPv4 1020083637
>>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 392u IPv4 1020083641
>>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 393u IPv4 1020083668
>>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 394u IPv4 1020083675
>>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 395u IPv4 1020083678
>>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 396u IPv4 1020083690
>>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 397u IPv4 1020083693
>>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 398u IPv4 1020083703
>>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 399u IPv4 1020083706
>>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 400u IPv4 1020083713
>>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 401u IPv4 1020083716
>>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 402u IPv4 1020083728
>>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 403u IPv4 1020083731
>>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 404u IPv4 1020083740
>>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 405u IPv4 1020083743
>>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 406u IPv4 1020083753
>>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 407u IPv4 1020083756
>>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 408u IPv4 1020083769
>>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 409u IPv4 1020083772
>>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 410u IPv4 1020083781
>>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 411u IPv4 1020083784
>>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 412u IPv4 1020083792
>>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 413u IPv4 1020083795
>>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 414u IPv4 1020083808
>>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 415u IPv4 1020083811
>>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 416u IPv4 1020083820
>>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 417u IPv4 1020083823
>>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 418u IPv4 1020083831
>>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 419u IPv4 1020083834
>>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 420u IPv4 1020083846
>>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 421u IPv4 1020083849
>>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 422u IPv4 1020083865
>>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 423u IPv4 1020083862
>>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 424u IPv4 1020083877
>>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 425u IPv4 1020083880
>>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 426u IPv4 1020083888
>>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 427u IPv4 1020083896
>>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 433u IPv4 1020083914
>>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 434u IPv4 1020083916
>>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 435u IPv4 1020083926
>>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 436u IPv4 1020083929
>>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 437u IPv4 1020084012
>>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 438u IPv4 1020083947
>>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 439u IPv4 1020083975
>>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 443u IPv4 1020083950
>>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 444u IPv4 1020083959
>>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 445u IPv4 1020083962
>>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 446u IPv4 1020083977
>>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 447u IPv4 1020083984
>>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 448u IPv4 1020088663
>>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 449u IPv4 1020083991
>>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 450u IPv4 1020088678
>>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 451u IPv4 1020088692
>>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 452u IPv4 1020088694
>>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 453u IPv4 1020092163
>>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>> java 5888 dwh 454u IPv4 1020092165
>>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>>
>>>>> Andy Kent wrote:
>>>>>
>>>>>
>>>>>> I have included the output from lsof, Unfortunately though I only
>>>>>> restarted the server a few mins ago and so it doesn't really
>>>>>> illustrate
>>>>>> the
>>>>>> problem. I'll try to post back with the same command tomorrow so that
>>>>>> we
>>>>>> can
>>>>>> diff between the two.
>>>>>>
>>>>>> Thanks, Andy.
>>>>>>
>>>>>> java 16908 hadoop cwd DIR 0,17 240
>>>>>> 7037865 /var/run/hadoop-0.20
>>>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>>>> 2 /
>>>>>> java 16908 hadoop txt REG 8,1 50810
>>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>>> java 16908 hadoop mem REG 8,1 127296
>>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>>> java 16908 hadoop mem REG 8,1 69472
>>>>>> 3735429
>>>>>>
>>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>>> java 16908 hadoop mem REG 8,1 44257
>>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>>> java 16908 hadoop mem REG 8,1 80760
>>>>>> 1164352 /lib/libresolv-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 22856
>>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 1145980
>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>> java 16908 hadoop mem REG 8,1 38105
>>>>>> 3555614
>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>>> java 16908 hadoop mem REG 8,1 111901
>>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>>> java 16908 hadoop mem REG 8,1 392124
>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>> java 16908 hadoop mem REG 8,1 229928
>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>> java 16908 hadoop mem REG 8,1 724225
>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>> java 16908 hadoop mem REG 8,1 169666
>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 45946
>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>> java 16908 hadoop mem REG 8,1 87325
>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>> java 16908 hadoop mem REG 8,1 179508
>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>> java 16908 hadoop mem REG 8,1 20202
>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>> java 16908 hadoop mem REG 8,1 95488
>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>> java 16908 hadoop mem REG 8,1 390114
>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>> java 16908 hadoop mem REG 8,1 500917
>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>> java 16908 hadoop mem REG 8,1 31931
>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>> java 16908 hadoop mem REG 8,1 19349
>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>> java 16908 hadoop mem REG 8,1 1798010
>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>> java 16908 hadoop mem REG 8,1 11412
>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>> java 16908 hadoop mem REG 8,1 14099
>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>> java 16908 hadoop mem REG 8,1 12641
>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>> java 16908 hadoop mem REG 8,1 7092
>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>> java 16908 hadoop mem REG 8,1 2446767
>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>> java 16908 hadoop mem REG 8,1 1154170
>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>> java 16908 hadoop mem REG 8,1 176174
>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>> java 16908 hadoop mem REG 8,1 1923209
>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>> java 16908 hadoop mem REG 8,1 261809
>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>> java 16908 hadoop mem REG 8,1 575389
>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>> java 16908 hadoop mem REG 8,1 258337
>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>> java 16908 hadoop mem REG 8,1 43035
>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 105209
>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 134910
>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 1024681
>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 15010
>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>> java 16908 hadoop mem REG 8,1 8601
>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>> java 16908 hadoop mem REG 8,1 15345
>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>> java 16908 hadoop mem REG 8,1 132368
>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>> java 16908 hadoop mem REG 8,1 65261
>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>> java 16908 hadoop mem REG 8,1 540852
>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>> java 16908 hadoop mem REG 8,1 73374
>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>> java 16908 hadoop mem REG 8,1 11428
>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>> java 16908 hadoop mem REG 8,1 198945
>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 163121
>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>> java 16908 hadoop mem REG 8,1 516429
>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>> java 16908 hadoop mem REG 8,1 321806
>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 76698
>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>> java 16908 hadoop mem REG 8,1 405086
>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>> java 16908 hadoop mem REG 8,1 37210
>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>> java 16908 hadoop mem REG 8,1 18684
>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>> java 16908 hadoop mem REG 8,1 37157
>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>> java 16908 hadoop mem REG 8,1 3566844
>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 180792
>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>> java 16908 hadoop mem REG 8,1 279781
>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>> java 16908 hadoop mem REG 8,1 2771746
>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>> java 16908 hadoop mem REG 8,1 12588713
>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>> java 16908 hadoop mem REG 8,1 25700
>>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>>> java 16908 hadoop mem REG 8,1 254076
>>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>>> java 16908 hadoop mem REG 8,1 51669471
>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>> java 16908 hadoop mem REG 8,1 92117
>>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>>> java 16908 hadoop mem REG 8,1 47528
>>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 43480
>>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 35632
>>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 55850
>>>>>> 3555581
>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>>> java 16908 hadoop mem REG 8,1 93080
>>>>>> 1164343 /lib/libnsl-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 234713
>>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>>> java 16908 hadoop mem REG 8,1 66580
>>>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>>> java 16908 hadoop mem REG 8,1 35784
>>>>>> 1164353 /lib/librt-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 526560
>>>>>> 1164341 /lib/libm-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 10987257
>>>>>> 3555599
>>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>>> java 16908 hadoop mem REG 8,1 1436976
>>>>>> 1164337 /lib/libc-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 14624
>>>>>> 1164340 /lib/libdl-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 130224
>>>>>> 1164351 /lib/libpthread-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 127480
>>>>>> 1164334 /lib/ld-2.7.so
>>>>>> java 16908 hadoop mem REG 8,1 618950
>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>> java 16908 hadoop mem REG 8,1 112341
>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>> java 16908 hadoop mem REG 8,1 32768
>>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>>> java 16908 hadoop mem REG 8,1 48335
>>>>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>>> java 16908 hadoop mem REG 8,1 41123
>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>> java 16908 hadoop 0u CHR 136,2
>>>>>> 4 /dev/pts/2 (deleted)
>>>>>> java 16908 hadoop 1u CHR 136,2
>>>>>> 4 /dev/pts/2 (deleted)
>>>>>> java 16908 hadoop 2u CHR 136,2
>>>>>> 4 /dev/pts/2 (deleted)
>>>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>>> java 16908 hadoop 6r REG 8,1 41123
>>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>>> java 16908 hadoop 7r REG 8,1 46725
>>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>>> java 16908 hadoop 8r REG 8,1 112341
>>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>>> java 16908 hadoop 9r REG 8,1 279781
>>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>>> java 16908 hadoop 10r REG 8,1 38015
>>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>>> java 16908 hadoop 11r REG 8,1 26202
>>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>>> java 16908 hadoop 12r REG 8,1 180792
>>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>>> java 16908 hadoop 14r REG 8,1 37157
>>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>>> java 16908 hadoop 15r REG 8,1 18684
>>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>>> java 16908 hadoop 16r REG 8,1 37210
>>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>>> java 16908 hadoop 17r REG 8,1 706710
>>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>>> java 16908 hadoop 18r REG 8,1 706710
>>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>>> java 16908 hadoop 19r REG 8,1 405086
>>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>>> java 16908 hadoop 20r REG 8,1 76698
>>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>>> java 16908 hadoop 21r REG 8,1 321806
>>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>>> java 16908 hadoop 22r REG 8,1 516429
>>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>>> java 16908 hadoop 23r REG 8,1 163121
>>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>>> java 16908 hadoop 24r REG 8,1 121070
>>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>>> java 16908 hadoop 25r REG 8,1 198945
>>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>>> java 16908 hadoop 26r REG 8,1 11428
>>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>>> java 16908 hadoop 27r REG 8,1 114342
>>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>>> java 16908 hadoop 28r REG 8,1 73374
>>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>>> java 16908 hadoop 29r REG 8,1 391834
>>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>>> java 16908 hadoop 30r REG 8,1 540852
>>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>>> java 16908 hadoop 31r REG 8,1 65261
>>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>>> java 16908 hadoop 32r REG 8,1 132368
>>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>>> java 16908 hadoop 33r REG 8,1 15345
>>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>>> java 16908 hadoop 34r REG 8,1 8601
>>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>>> java 16908 hadoop 35r REG 8,1 15010
>>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>>> java 16908 hadoop 37r REG 8,1 134910
>>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>>> java 16908 hadoop 38r REG 8,1 105209
>>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>>> java 16908 hadoop 39r REG 8,1 43035
>>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>>> java 16908 hadoop 40r REG 8,1 258337
>>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>>> java 16908 hadoop 41r REG 8,1 46725
>>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>>> java 16908 hadoop 42r REG 8,1 575389
>>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>>> java 16908 hadoop 43r REG 8,1 261809
>>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>>> java 16908 hadoop 44r REG 8,1 38015
>>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>>> java 16908 hadoop 45r REG 8,1 26202
>>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>>> java 16908 hadoop 47r REG 8,1 176174
>>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>>> java 16908 hadoop 50r REG 8,1 7092
>>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>>> java 16908 hadoop 51r REG 8,1 12641
>>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>>> java 16908 hadoop 52r REG 8,1 14099
>>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>>> java 16908 hadoop 53r REG 8,1 11412
>>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>>> java 16908 hadoop 55r REG 8,1 19349
>>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>>> java 16908 hadoop 56r REG 8,1 31931
>>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>>> java 16908 hadoop 57r REG 8,1 500917
>>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>>> java 16908 hadoop 58r REG 8,1 390114
>>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>>> java 16908 hadoop 59r REG 8,1 95488
>>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>>> java 16908 hadoop 60r REG 8,1 20202
>>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>>> java 16908 hadoop 61r REG 8,1 179508
>>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>>> java 16908 hadoop 62r REG 8,1 87325
>>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>>> java 16908 hadoop 63r REG 8,1 45946
>>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>>> java 16908 hadoop 64r REG 8,1 121070
>>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>>> java 16908 hadoop 65r REG 8,1 114342
>>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>>> java 16908 hadoop 66r REG 8,1 169666
>>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>>> java 16908 hadoop 67r REG 8,1 391834
>>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>>> java 16908 hadoop 68r REG 8,1 724225
>>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>>> java 16908 hadoop 69r REG 8,1 229928
>>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>>> java 16908 hadoop 70r REG 8,1 392124
>>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>>> java 16908 hadoop 71r CHR 1,8
>>>>>> 1818 /dev/random
>>>>>> java 16908 hadoop 72r CHR 1,9
>>>>>> 1824 /dev/urandom
>>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>>> TCP *:webmin (LISTEN)
>>>>>> java 16908 hadoop 74u sock 0,4
>>>>>> 153142739 can't identify protocol
>>>>>> java 16908 hadoop 76w REG 8,1 6867
>>>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 81w REG 8,1 15491
>>>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 83r REG 8,1 618950
>>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>>> 153143422 socket
>>>>>> java 16908 hadoop 86w REG 8,1 62510
>>>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>>> java 16908 hadoop 87w REG 8,1 960
>>>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 113w REG 8,1 11562
>>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>>> 154318314 pipe
>>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (ESTABLISHED)
>>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>>> 154318314 pipe
>>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 249w REG 8,1 15230
>>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 253u 0000 0,6 0
>>>>>> 493 /anon_inode:[eventpoll]
>>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>>> TCP
>>>>>>
>>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>>> (CLOSE_WAIT)
>>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345"
>>>>>>> where
>>>>>>> 12345 is the process id of the hive server?
>>>>>>> We should be able to see the names of the files that are open.
>>>>>>>
>>>>>>> Zheng
>>>>>>>
>>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Nope, no luck so far.
>>>>>>>>
>>>>>>>> We have upped the number of file descriptors and are having to
>>>>>>>> restart
>>>>>>>> hive every week or so :(
>>>>>>>>
>>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>>
>>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>>
>>>>>>>>> Andy Kent wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I can give try and give it a go. I'm not convinced though as we
>>>>>>>>>> are
>>>>>>>>>> working with CSV files and don't touch sequence files at all at
>>>>>>>>>> the
>>>>>>>>>> moment.
>>>>>>>>>>
>>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>>>>> Hive 0.40
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>>> filehandle
>>>>>>>>>>> open and never close it.
>>>>>>>>>>>
>>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get
>>>>>>>>>>> some
>>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>>>> this to
>>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>>>>> combination.. maybe if you're doing queries and then closing
>>>>>>>>>>> your
>>>>>>>>>>> thrift
>>>>>>>>>>> connection before reading all results, Hive doesn't know what to
>>>>>>>>>>> do
>>>>>>>>>>> and
>>>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>>>> they might
>>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting
>>>>>>>>>>>> to
>>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and
>>>>>>>>>>>> not
>>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>
>>>>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby
>>>>>>>>>>> code
>>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>>
>>>>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>>>> should post
>>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>>> haven't noticed
>>>>>>>>>>> this?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Yours,
>>>>>>> Zheng
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
--
Sent from my mobile device
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
jstack on the Hive process:
2010-02-16
09:30:47
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
mode):
"Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
waiting on condition [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
condition [0x000000004238d000]
java.lang.Thread.State: WAITING
(parking)
at sun.misc.Unsafe.$$YJP$$park(Native
Method)
- parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
sun.misc.Unsafe.park(Unsafe.java)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
condition [0x0000000042690000]
java.lang.Thread.State: WAITING
(parking)
at sun.misc.Unsafe.$$YJP$$park(Native
Method)
- parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
sun.misc.Unsafe.park(Unsafe.java)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
"LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
on condition [0x0000000042791000]
java.lang.Thread.State: TIMED_WAITING
(sleeping)
at java.lang.Thread.$$YJP$$sleep(Native
Method)
at
java.lang.Thread.sleep(Thread.java)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
condition [0x000000004258f000]
java.lang.Thread.State: WAITING
(parking)
at sun.misc.Unsafe.$$YJP$$park(Native
Method)
- parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
sun.misc.Unsafe.park(Unsafe.java)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
condition [0x000000004248e000]
java.lang.Thread.State: WAITING
(parking)
at sun.misc.Unsafe.$$YJP$$park(Native
Method)
- parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
sun.misc.Unsafe.park(Unsafe.java)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x00007fbe96883800 nid=0x183b waiting on
condition [0x0000000040c7e000]
java.lang.Thread.State: WAITING
(parking)
at sun.misc.Unsafe.$$YJP$$park(Native
Method)
- parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
sun.misc.Unsafe.park(Unsafe.java)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
"YJPAgent-RequestListener" daemon prio=10 tid=0x00007fbe96875800
nid=0x179d runnable [0x000000004228c000]
java.lang.Thread.State:
RUNNABLE
at java.net.SocketInputStream.socketRead0(Native
Method)
at
java.net.SocketInputStream.read(SocketInputStream.java:129)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00007fbf10922580> (a
java.io.BufferedInputStream)
at
java.io.DataInputStream.readFully(DataInputStream.java:178)
at
java.io.DataInputStream.readLong(DataInputStream.java:399)
at
com.yourkit.runtime.Packet.readHeader(Packet.java:40)
at
com.yourkit.runtime.RequestProcessing.dispatchRequest(RequestProcessing.java:16)
at
com.yourkit.runtime.Core$4.run(Core.java:525)
at
java.lang.Thread.run(Thread.java:619)
"YJPAgent-Telemetry" daemon prio=10 tid=0x00007fbe96c05800 nid=0x179c
waiting on condition [0x0000000040d7f000]
java.lang.Thread.State: TIMED_WAITING
(sleeping)
at java.lang.Thread.$$YJP$$sleep(Native
Method)
at
java.lang.Thread.sleep(Thread.java)
at
com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
"Low Memory Detector" daemon prio=10 tid=0x00007fbe9067e800 nid=0x1798
runnable [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007fbe9067d000 nid=0x1797
waiting on condition [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00007fbe9067c000 nid=0x1796
waiting on condition [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00007fbe905f1800
nid=0x1795 runnable [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"YJPAgent-CPUSampler" daemon prio=10 tid=0x00007fbe9008f000 nid=0x1794
runnable [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fbe9009e800 nid=0x1792
runnable [0x0000000000000000]
java.lang.Thread.State:
RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fbe90051800 nid=0x1791 in
Object.wait() [0x000000004119d000]
java.lang.Thread.State: WAITING (on object
monitor)
at java.lang.Object.wait(Native
Method)
- waiting on <0x00007fbea52c6790> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007fbea52c6790> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007fbe9009d000 nid=0x1790 in
Object.wait() [0x000000004208a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007fbea52c6450> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000040265000 nid=0x1781 runnable
[0x00000000414e3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x00007fbea52c6868> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
at
org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
"VM Thread" prio=10 tid=0x00007fbe9006c000 nid=0x178f runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004026f800
nid=0x1782 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040271000
nid=0x1783 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040273000
nid=0x1784 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040275000
nid=0x1785 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040276800
nid=0x1786 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040278800
nid=0x1787 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x000000004027a800
nid=0x1788 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x000000004027c000
nid=0x1789 runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x000000004027e000
nid=0x178a runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040280000
nid=0x178b runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040281800
nid=0x178c runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000040283800
nid=0x178d runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000040285800
nid=0x178e runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fbe9064a000 nid=0x1799
waiting on condition
JNI global references: 375194
Zheng Shao wrote:
> It seems that Hive server is leaking hdfs handles.
> Not sure if hadoop has a command to show open hdfs handles.
>
> Can you also do a jstack of the Hive Server process?
> First, "ps -elf | grep java" to find the hive server process id, and
> the jdk directory
> Second, "<jdk directory>/bin/jstack <pid>".
>
> This will tell us if there are clients of the Hive Server that didn't
> close their connections yet.
>
> Zheng
>
> On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>
>> That's the datanode port. So Hdfs. Any trick to see what is opened on hdfs?
>> like an lsof for hdfs :)
>>
>> Zheng Shao wrote:
>>
>>> What's running at victoria.ebuddy.com:50010 ?
>>>
>>> Zhemg
>>>
>>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>>
>>>> I think it's lines like this:
>>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>>
>>>> We have the same.
>>>> Currently no queries running and many of those connections open. After
>>>> running some queries a few more are added to the list.
>>>>
>>>> java 5888 dwh 389u IPv4 1020083623
>>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 390u IPv4 1020083626
>>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 391u IPv4 1020083637
>>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 392u IPv4 1020083641
>>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 393u IPv4 1020083668
>>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 394u IPv4 1020083675
>>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 395u IPv4 1020083678
>>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 396u IPv4 1020083690
>>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 397u IPv4 1020083693
>>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 398u IPv4 1020083703
>>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 399u IPv4 1020083706
>>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 400u IPv4 1020083713
>>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 401u IPv4 1020083716
>>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 402u IPv4 1020083728
>>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 403u IPv4 1020083731
>>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 404u IPv4 1020083740
>>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 405u IPv4 1020083743
>>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 406u IPv4 1020083753
>>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 407u IPv4 1020083756
>>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 408u IPv4 1020083769
>>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 409u IPv4 1020083772
>>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 410u IPv4 1020083781
>>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 411u IPv4 1020083784
>>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 412u IPv4 1020083792
>>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 413u IPv4 1020083795
>>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 414u IPv4 1020083808
>>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 415u IPv4 1020083811
>>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 416u IPv4 1020083820
>>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 417u IPv4 1020083823
>>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 418u IPv4 1020083831
>>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 419u IPv4 1020083834
>>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 420u IPv4 1020083846
>>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 421u IPv4 1020083849
>>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 422u IPv4 1020083865
>>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 423u IPv4 1020083862
>>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 424u IPv4 1020083877
>>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 425u IPv4 1020083880
>>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 426u IPv4 1020083888
>>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 427u IPv4 1020083896
>>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 433u IPv4 1020083914
>>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 434u IPv4 1020083916
>>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 435u IPv4 1020083926
>>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 436u IPv4 1020083929
>>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 437u IPv4 1020084012
>>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 438u IPv4 1020083947
>>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 439u IPv4 1020083975
>>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 443u IPv4 1020083950
>>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 444u IPv4 1020083959
>>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 445u IPv4 1020083962
>>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 446u IPv4 1020083977
>>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 447u IPv4 1020083984
>>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 448u IPv4 1020088663
>>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 449u IPv4 1020083991
>>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 450u IPv4 1020088678
>>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 451u IPv4 1020088692
>>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 452u IPv4 1020088694
>>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 453u IPv4 1020092163
>>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>> java 5888 dwh 454u IPv4 1020092165
>>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>>
>>>> Andy Kent wrote:
>>>>
>>>>
>>>>> I have included the output from lsof, Unfortunately though I only
>>>>> restarted the server a few mins ago and so it doesn't really illustrate
>>>>> the
>>>>> problem. I'll try to post back with the same command tomorrow so that we
>>>>> can
>>>>> diff between the two.
>>>>>
>>>>> Thanks, Andy.
>>>>>
>>>>> java 16908 hadoop cwd DIR 0,17 240
>>>>> 7037865 /var/run/hadoop-0.20
>>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>>> 2 /
>>>>> java 16908 hadoop txt REG 8,1 50810
>>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>>> java 16908 hadoop mem REG 8,1 127296
>>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>>> java 16908 hadoop mem REG 8,1 69472
>>>>> 3735429
>>>>>
>>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>>> java 16908 hadoop mem REG 8,1 44257
>>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>>> java 16908 hadoop mem REG 8,1 80760
>>>>> 1164352 /lib/libresolv-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 22856
>>>>> 1164345 /lib/libnss_dns-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 1145980
>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>> java 16908 hadoop mem REG 8,1 38105
>>>>> 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>>> java 16908 hadoop mem REG 8,1 111901
>>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>>> java 16908 hadoop mem REG 8,1 392124
>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>> java 16908 hadoop mem REG 8,1 229928
>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>> java 16908 hadoop mem REG 8,1 724225
>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>> java 16908 hadoop mem REG 8,1 169666
>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>> java 16908 hadoop mem REG 8,1 45946
>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>> java 16908 hadoop mem REG 8,1 87325
>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>> java 16908 hadoop mem REG 8,1 179508
>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>> java 16908 hadoop mem REG 8,1 20202
>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>> java 16908 hadoop mem REG 8,1 95488
>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>> java 16908 hadoop mem REG 8,1 390114
>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>> java 16908 hadoop mem REG 8,1 500917
>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>> java 16908 hadoop mem REG 8,1 31931
>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>> java 16908 hadoop mem REG 8,1 19349
>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>> java 16908 hadoop mem REG 8,1 1798010
>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>> java 16908 hadoop mem REG 8,1 11412
>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>> java 16908 hadoop mem REG 8,1 14099
>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>> java 16908 hadoop mem REG 8,1 12641
>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>> java 16908 hadoop mem REG 8,1 7092
>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>> java 16908 hadoop mem REG 8,1 2446767
>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>> java 16908 hadoop mem REG 8,1 1154170
>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>> java 16908 hadoop mem REG 8,1 176174
>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>> java 16908 hadoop mem REG 8,1 1923209
>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>> java 16908 hadoop mem REG 8,1 261809
>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>> java 16908 hadoop mem REG 8,1 575389
>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>> java 16908 hadoop mem REG 8,1 258337
>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>> java 16908 hadoop mem REG 8,1 43035
>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>> java 16908 hadoop mem REG 8,1 105209
>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>> java 16908 hadoop mem REG 8,1 134910
>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>> java 16908 hadoop mem REG 8,1 1024681
>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>> java 16908 hadoop mem REG 8,1 15010
>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>> java 16908 hadoop mem REG 8,1 8601
>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>> java 16908 hadoop mem REG 8,1 15345
>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>> java 16908 hadoop mem REG 8,1 132368
>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>> java 16908 hadoop mem REG 8,1 65261
>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>> java 16908 hadoop mem REG 8,1 540852
>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>> java 16908 hadoop mem REG 8,1 391834
>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>> java 16908 hadoop mem REG 8,1 73374
>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>> java 16908 hadoop mem REG 8,1 114342
>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>> java 16908 hadoop mem REG 8,1 11428
>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>> java 16908 hadoop mem REG 8,1 198945
>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>> java 16908 hadoop mem REG 8,1 121070
>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>> java 16908 hadoop mem REG 8,1 163121
>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>> java 16908 hadoop mem REG 8,1 516429
>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>> java 16908 hadoop mem REG 8,1 321806
>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>> java 16908 hadoop mem REG 8,1 76698
>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>> java 16908 hadoop mem REG 8,1 405086
>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>> java 16908 hadoop mem REG 8,1 706710
>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>> java 16908 hadoop mem REG 8,1 37210
>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>> java 16908 hadoop mem REG 8,1 18684
>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>> java 16908 hadoop mem REG 8,1 37157
>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>> java 16908 hadoop mem REG 8,1 3566844
>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>> java 16908 hadoop mem REG 8,1 180792
>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>> java 16908 hadoop mem REG 8,1 26202
>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>> java 16908 hadoop mem REG 8,1 38015
>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>> java 16908 hadoop mem REG 8,1 279781
>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>> java 16908 hadoop mem REG 8,1 2771746
>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>> java 16908 hadoop mem REG 8,1 12588713
>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>> java 16908 hadoop mem REG 8,1 25700
>>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>>> java 16908 hadoop mem REG 8,1 254076
>>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>>> java 16908 hadoop mem REG 8,1 51669471
>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>> java 16908 hadoop mem REG 8,1 92117
>>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>>> java 16908 hadoop mem REG 8,1 47528
>>>>> 1164346 /lib/libnss_files-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 43480
>>>>> 1164348 /lib/libnss_nis-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 35632
>>>>> 1164344 /lib/libnss_compat-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 55850
>>>>> 3555581
>>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>>> java 16908 hadoop mem REG 8,1 93080
>>>>> 1164343 /lib/libnsl-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 234713
>>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>>> java 16908 hadoop mem REG 8,1 66580
>>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>>> java 16908 hadoop mem REG 8,1 35784
>>>>> 1164353 /lib/librt-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 526560
>>>>> 1164341 /lib/libm-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 10987257
>>>>> 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>>> java 16908 hadoop mem REG 8,1 1436976
>>>>> 1164337 /lib/libc-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 14624
>>>>> 1164340 /lib/libdl-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 130224
>>>>> 1164351 /lib/libpthread-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 127480
>>>>> 1164334 /lib/ld-2.7.so
>>>>> java 16908 hadoop mem REG 8,1 618950
>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>> java 16908 hadoop mem REG 8,1 112341
>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>> java 16908 hadoop mem REG 8,1 46725
>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>> java 16908 hadoop mem REG 8,1 32768
>>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>>> java 16908 hadoop mem REG 8,1 48335
>>>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>>> java 16908 hadoop mem REG 8,1 41123
>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>> java 16908 hadoop 0u CHR 136,2
>>>>> 4 /dev/pts/2 (deleted)
>>>>> java 16908 hadoop 1u CHR 136,2
>>>>> 4 /dev/pts/2 (deleted)
>>>>> java 16908 hadoop 2u CHR 136,2
>>>>> 4 /dev/pts/2 (deleted)
>>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>>> java 16908 hadoop 6r REG 8,1 41123
>>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>>> java 16908 hadoop 7r REG 8,1 46725
>>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>>> java 16908 hadoop 8r REG 8,1 112341
>>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>>> java 16908 hadoop 9r REG 8,1 279781
>>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>>> java 16908 hadoop 10r REG 8,1 38015
>>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>>> java 16908 hadoop 11r REG 8,1 26202
>>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>>> java 16908 hadoop 12r REG 8,1 180792
>>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>>> java 16908 hadoop 14r REG 8,1 37157
>>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>>> java 16908 hadoop 15r REG 8,1 18684
>>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>>> java 16908 hadoop 16r REG 8,1 37210
>>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>>> java 16908 hadoop 17r REG 8,1 706710
>>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>>> java 16908 hadoop 18r REG 8,1 706710
>>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>>> java 16908 hadoop 19r REG 8,1 405086
>>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>>> java 16908 hadoop 20r REG 8,1 76698
>>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>>> java 16908 hadoop 21r REG 8,1 321806
>>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>>> java 16908 hadoop 22r REG 8,1 516429
>>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>>> java 16908 hadoop 23r REG 8,1 163121
>>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>>> java 16908 hadoop 24r REG 8,1 121070
>>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>>> java 16908 hadoop 25r REG 8,1 198945
>>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>>> java 16908 hadoop 26r REG 8,1 11428
>>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>>> java 16908 hadoop 27r REG 8,1 114342
>>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>>> java 16908 hadoop 28r REG 8,1 73374
>>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>>> java 16908 hadoop 29r REG 8,1 391834
>>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>>> java 16908 hadoop 30r REG 8,1 540852
>>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>>> java 16908 hadoop 31r REG 8,1 65261
>>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>>> java 16908 hadoop 32r REG 8,1 132368
>>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>>> java 16908 hadoop 33r REG 8,1 15345
>>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>>> java 16908 hadoop 34r REG 8,1 8601
>>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>>> java 16908 hadoop 35r REG 8,1 15010
>>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>>> java 16908 hadoop 37r REG 8,1 134910
>>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>>> java 16908 hadoop 38r REG 8,1 105209
>>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>>> java 16908 hadoop 39r REG 8,1 43035
>>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>>> java 16908 hadoop 40r REG 8,1 258337
>>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>>> java 16908 hadoop 41r REG 8,1 46725
>>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>>> java 16908 hadoop 42r REG 8,1 575389
>>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>>> java 16908 hadoop 43r REG 8,1 261809
>>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>>> java 16908 hadoop 44r REG 8,1 38015
>>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>>> java 16908 hadoop 45r REG 8,1 26202
>>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>>> java 16908 hadoop 47r REG 8,1 176174
>>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>>> java 16908 hadoop 50r REG 8,1 7092
>>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>>> java 16908 hadoop 51r REG 8,1 12641
>>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>>> java 16908 hadoop 52r REG 8,1 14099
>>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>>> java 16908 hadoop 53r REG 8,1 11412
>>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>>> java 16908 hadoop 55r REG 8,1 19349
>>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>>> java 16908 hadoop 56r REG 8,1 31931
>>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>>> java 16908 hadoop 57r REG 8,1 500917
>>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>>> java 16908 hadoop 58r REG 8,1 390114
>>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>>> java 16908 hadoop 59r REG 8,1 95488
>>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>>> java 16908 hadoop 60r REG 8,1 20202
>>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>>> java 16908 hadoop 61r REG 8,1 179508
>>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>>> java 16908 hadoop 62r REG 8,1 87325
>>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>>> java 16908 hadoop 63r REG 8,1 45946
>>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>>> java 16908 hadoop 64r REG 8,1 121070
>>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>>> java 16908 hadoop 65r REG 8,1 114342
>>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>>> java 16908 hadoop 66r REG 8,1 169666
>>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>>> java 16908 hadoop 67r REG 8,1 391834
>>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>>> java 16908 hadoop 68r REG 8,1 724225
>>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>>> java 16908 hadoop 69r REG 8,1 229928
>>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>>> java 16908 hadoop 70r REG 8,1 392124
>>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>>> java 16908 hadoop 71r CHR 1,8
>>>>> 1818 /dev/random
>>>>> java 16908 hadoop 72r CHR 1,9
>>>>> 1824 /dev/urandom
>>>>> java 16908 hadoop 73u IPv6 153142741
>>>>> TCP *:webmin (LISTEN)
>>>>> java 16908 hadoop 74u sock 0,4
>>>>> 153142739 can't identify protocol
>>>>> java 16908 hadoop 76w REG 8,1 6867
>>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>>> java 16908 hadoop 77u IPv6 154300915
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>>> java 16908 hadoop 79u IPv6 153477399
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 80u IPv6 153594480
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 81w REG 8,1 15491
>>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>>> java 16908 hadoop 82u IPv6 153177590
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 83r REG 8,1 618950
>>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>>> java 16908 hadoop 84u IPv6 153177591
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>>> 153143422 socket
>>>>> java 16908 hadoop 86w REG 8,1 62510
>>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>>> java 16908 hadoop 87w REG 8,1 960
>>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>>> java 16908 hadoop 88u IPv6 153219191
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 89u IPv6 153143432
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 90u IPv6 153143434
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 91u IPv6 153219192
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 92u IPv6 153477400
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 93u IPv6 153445420
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 94u IPv6 153385926
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 95u IPv6 153322551
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 96u IPv6 153220498
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 97u IPv6 153220499
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 98u IPv6 153273816
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 99u IPv6 153273819
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 100u IPv6 153293338
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 102u IPv6 153708229
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 103u IPv6 153276086
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 104u IPv6 153276087
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 105u IPv6 153291015
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 106u IPv6 153291016
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 107u IPv6 153293339
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 108u IPv6 153291444
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 109u IPv6 153291445
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 110u IPv6 153294552
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 111u IPv6 153294209
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 112u IPv6 153294210
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 113w REG 8,1 11562
>>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>>> java 16908 hadoop 114u IPv6 153294232
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 115u IPv6 153294233
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 116u IPv6 153294553
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 117u IPv6 153308598
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 118u IPv6 153299396
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 119u IPv6 153299397
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 120u IPv6 153305498
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 121u IPv6 153305501
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 122u IPv6 153322053
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 123u IPv6 153306847
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 124u IPv6 153306848
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 125u IPv6 153308599
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 126u IPv6 153308072
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 127u IPv6 153308073
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 128u IPv6 153322552
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 129u IPv6 153318287
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 130u IPv6 153318288
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 131u IPv6 153322054
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 132u IPv6 153321874
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 133u IPv6 153321875
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 134u IPv6 153708231
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 135u IPv6 153322490
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 136u IPv6 153322491
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 137u IPv6 153330756
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 138u IPv6 153322687
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 139u IPv6 153322688
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 140u IPv6 153325417
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 141u IPv6 153322873
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 142u IPv6 153322874
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 143u IPv6 153325418
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 144u IPv6 153355133
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 145u IPv6 153325537
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 146u IPv6 153325538
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 147u IPv6 153330757
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 148u IPv6 153330673
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 149u IPv6 153330674
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 150u IPv6 153342009
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 151u IPv6 153342010
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 152u IPv6 153343873
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 153u IPv6 153342462
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 154u IPv6 153342463
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 155u IPv6 153343874
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 156u IPv6 153348641
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 157u IPv6 153344781
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 158u IPv6 153344782
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 159u IPv6 153348642
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 160u IPv6 153354839
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 161u IPv6 153353725
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 162u IPv6 153353726
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 163u IPv6 153354840
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 164u IPv6 153363372
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 165u IPv6 153354979
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 166u IPv6 153354980
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 167u IPv6 153355134
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 168u IPv6 153385927
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 169u IPv6 153445421
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 170u IPv6 153405211
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 171u IPv6 153363371
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 172u IPv6 153405212
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 173u IPv6 153373260
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 174u IPv6 153460225
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 175u IPv6 153449798
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 176u IPv6 153360445
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 177u IPv6 153360459
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 178u IPv6 153449799
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 179u IPv6 153373262
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 180u IPv6 153509124
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 181u IPv6 153411087
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 182u IPv6 153411089
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 183u IPv6 153460228
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 184u IPv6 153428120
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 185u IPv6 153428121
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 187u IPv6 153600041
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 188u IPv6 153557843
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 189u IPv6 153462763
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 190u IPv6 153460417
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 191u IPv6 153460418
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 192u IPv6 153462764
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 193u IPv6 153471908
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 194u IPv6 153468521
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 195u IPv6 153468522
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 196u IPv6 153471980
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 197u IPv6 153557844
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 198u IPv6 153476603
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 199u IPv6 153476604
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 200u IPv6 153508848
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 201u IPv6 153507533
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 202u IPv6 153507534
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 203u IPv6 153508419
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 204u IPv6 153508234
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 205u IPv6 153508235
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 206u IPv6 153508422
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 207u IPv6 153509125
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 209u IPv6 153508845
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 210u IPv6 153508846
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 211u IPv6 153572667
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 212u IPv6 153508849
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 213u IPv6 153516692
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 214u IPv6 153516695
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 216u IPv6 153610112
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 217u IPv6 153610114
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 218u IPv6 153557944
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 219u IPv6 153557945
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 220u IPv6 153572668
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 221u IPv6 153585054
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 222u IPv6 153582445
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 223u IPv6 153582447
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 224u IPv6 153585055
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 225u IPv6 153708561
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 226u IPv6 153594476
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 227u IPv6 153638916
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 229u IPv6 153708562
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 230u IPv6 153600134
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 231u IPv6 153638922
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 232u IPv6 153635793
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 233u IPv6 153708619
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 234u IPv6 153703803
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 235u IPv6 153703804
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 236u IPv6 153604398
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 237u IPv6 153604506
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 238u IPv6 153708578
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 241u IPv6 153635795
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 244r FIFO 0,5
>>>>> 154318314 pipe
>>>>> java 16908 hadoop 245u IPv6 153706355
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 246u IPv6 153706356
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (ESTABLISHED)
>>>>> java 16908 hadoop 247w FIFO 0,5
>>>>> 154318314 pipe
>>>>> java 16908 hadoop 248u IPv6 153708579
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 249w REG 8,1 15230
>>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>>> java 16908 hadoop 250u IPv6 153708593
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 251u IPv6 153708594
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 252u IPv6 153708620
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 253u 0000 0,6 0
>>>>> 493 /anon_inode:[eventpoll]
>>>>> java 16908 hadoop 255u IPv6 153742520
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 256u IPv6 153742521
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 259u IPv6 153742533
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 260u IPv6 153742534
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 262u IPv6 153742547
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 263u IPv6 153742548
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 265u IPv6 153742562
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> java 16908 hadoop 266u IPv6 153742563
>>>>> TCP
>>>>>
>>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>>> (CLOSE_WAIT)
>>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
>>>>>> 12345 is the process id of the hive server?
>>>>>> We should be able to see the names of the files that are open.
>>>>>>
>>>>>> Zheng
>>>>>>
>>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Nope, no luck so far.
>>>>>>>
>>>>>>> We have upped the number of file descriptors and are having to restart
>>>>>>> hive every week or so :(
>>>>>>>
>>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>>
>>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>>
>>>>>>>> Andy Kent wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I can give try and give it a go. I'm not convinced though as we are
>>>>>>>>> working with CSV files and don't touch sequence files at all at the
>>>>>>>>> moment.
>>>>>>>>>
>>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>>>> Hive 0.40
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>>> filehandle
>>>>>>>>>> open and never close it.
>>>>>>>>>>
>>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some
>>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>>> this to
>>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>>>> combination.. maybe if you're doing queries and then closing your
>>>>>>>>>> thrift
>>>>>>>>>> connection before reading all results, Hive doesn't know what to do
>>>>>>>>>> and
>>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>>> they might
>>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting
>>>>>>>>>>> to
>>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and
>>>>>>>>>>> not
>>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Thanks for the response.
>>>>>>>>>>
>>>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby
>>>>>>>>>> code
>>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>>
>>>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>>> should post
>>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>>> haven't noticed
>>>>>>>>>> this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Yours,
>>>>>> Zheng
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>>
>>
>
>
>
> --
> Yours,
> Zheng
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
It seems that Hive server is leaking hdfs handles.
Not sure if hadoop has a command to show open hdfs handles.
Can you also do a jstack of the Hive Server process?
First, "ps -elf | grep java" to find the hive server process id, and
the jdk directory
Second, "<jdk directory>/bin/jstack <pid>".
This will tell us if there are clients of the Hive Server that didn't
close their connections yet.
Zheng
On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <bs...@ebuddy.com> wrote:
> That's the datanode port. So Hdfs. Any trick to see what is opened on hdfs?
> like an lsof for hdfs :)
>
> Zheng Shao wrote:
>>
>> What's running at victoria.ebuddy.com:50010 ?
>>
>> Zhemg
>>
>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>>>
>>> I think it's lines like this:
>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>
>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>>
>>> We have the same.
>>> Currently no queries running and many of those connections open. After
>>> running some queries a few more are added to the list.
>>>
>>> java 5888 dwh 389u IPv4 1020083623
>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 390u IPv4 1020083626
>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 391u IPv4 1020083637
>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 392u IPv4 1020083641
>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 393u IPv4 1020083668
>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 394u IPv4 1020083675
>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 395u IPv4 1020083678
>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 396u IPv4 1020083690
>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 397u IPv4 1020083693
>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 398u IPv4 1020083703
>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 399u IPv4 1020083706
>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 400u IPv4 1020083713
>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 401u IPv4 1020083716
>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 402u IPv4 1020083728
>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 403u IPv4 1020083731
>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 404u IPv4 1020083740
>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 405u IPv4 1020083743
>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 406u IPv4 1020083753
>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 407u IPv4 1020083756
>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 408u IPv4 1020083769
>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 409u IPv4 1020083772
>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 410u IPv4 1020083781
>>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 411u IPv4 1020083784
>>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 412u IPv4 1020083792
>>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 413u IPv4 1020083795
>>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 414u IPv4 1020083808
>>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 415u IPv4 1020083811
>>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 416u IPv4 1020083820
>>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 417u IPv4 1020083823
>>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 418u IPv4 1020083831
>>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 419u IPv4 1020083834
>>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 420u IPv4 1020083846
>>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 421u IPv4 1020083849
>>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 422u IPv4 1020083865
>>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 423u IPv4 1020083862
>>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 424u IPv4 1020083877
>>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 425u IPv4 1020083880
>>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 426u IPv4 1020083888
>>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 427u IPv4 1020083896
>>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 433u IPv4 1020083914
>>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 434u IPv4 1020083916
>>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 435u IPv4 1020083926
>>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 436u IPv4 1020083929
>>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 437u IPv4 1020084012
>>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 438u IPv4 1020083947
>>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 439u IPv4 1020083975
>>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 443u IPv4 1020083950
>>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 444u IPv4 1020083959
>>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 445u IPv4 1020083962
>>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 446u IPv4 1020083977
>>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 447u IPv4 1020083984
>>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 448u IPv4 1020088663
>>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 449u IPv4 1020083991
>>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 450u IPv4 1020088678
>>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 451u IPv4 1020088692
>>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 452u IPv4 1020088694
>>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 453u IPv4 1020092163
>>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 454u IPv4 1020092165
>>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>>
>>> Andy Kent wrote:
>>>
>>>>
>>>> I have included the output from lsof, Unfortunately though I only
>>>> restarted the server a few mins ago and so it doesn't really illustrate
>>>> the
>>>> problem. I'll try to post back with the same command tomorrow so that we
>>>> can
>>>> diff between the two.
>>>>
>>>> Thanks, Andy.
>>>>
>>>> java 16908 hadoop cwd DIR 0,17 240
>>>> 7037865 /var/run/hadoop-0.20
>>>> java 16908 hadoop rtd DIR 8,1 4096
>>>> 2 /
>>>> java 16908 hadoop txt REG 8,1 50810
>>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>>> java 16908 hadoop mem REG 8,1 127296
>>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>>> java 16908 hadoop mem REG 8,1 69472
>>>> 3735429
>>>>
>>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>>> java 16908 hadoop mem REG 8,1 44257
>>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>>> java 16908 hadoop mem REG 8,1 80760
>>>> 1164352 /lib/libresolv-2.7.so
>>>> java 16908 hadoop mem REG 8,1 22856
>>>> 1164345 /lib/libnss_dns-2.7.so
>>>> java 16908 hadoop mem REG 8,1 1145980
>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>> java 16908 hadoop mem REG 8,1 38105
>>>> 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>>> java 16908 hadoop mem REG 8,1 111901
>>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>>> java 16908 hadoop mem REG 8,1 392124
>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>> java 16908 hadoop mem REG 8,1 229928
>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>> java 16908 hadoop mem REG 8,1 724225
>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>> java 16908 hadoop mem REG 8,1 391834
>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>> java 16908 hadoop mem REG 8,1 169666
>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>> java 16908 hadoop mem REG 8,1 114342
>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>> java 16908 hadoop mem REG 8,1 121070
>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>> java 16908 hadoop mem REG 8,1 45946
>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>> java 16908 hadoop mem REG 8,1 87325
>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>> java 16908 hadoop mem REG 8,1 179508
>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>> java 16908 hadoop mem REG 8,1 20202
>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>> java 16908 hadoop mem REG 8,1 95488
>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>> java 16908 hadoop mem REG 8,1 390114
>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>> java 16908 hadoop mem REG 8,1 500917
>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>> java 16908 hadoop mem REG 8,1 31931
>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>> java 16908 hadoop mem REG 8,1 19349
>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>> java 16908 hadoop mem REG 8,1 1798010
>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>> java 16908 hadoop mem REG 8,1 11412
>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>> java 16908 hadoop mem REG 8,1 14099
>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>> java 16908 hadoop mem REG 8,1 12641
>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>> java 16908 hadoop mem REG 8,1 7092
>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>> java 16908 hadoop mem REG 8,1 2446767
>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>> java 16908 hadoop mem REG 8,1 1154170
>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>> java 16908 hadoop mem REG 8,1 176174
>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>> java 16908 hadoop mem REG 8,1 1923209
>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>> java 16908 hadoop mem REG 8,1 26202
>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>> java 16908 hadoop mem REG 8,1 38015
>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>> java 16908 hadoop mem REG 8,1 261809
>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>> java 16908 hadoop mem REG 8,1 575389
>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>> java 16908 hadoop mem REG 8,1 46725
>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>> java 16908 hadoop mem REG 8,1 258337
>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>> java 16908 hadoop mem REG 8,1 43035
>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>> java 16908 hadoop mem REG 8,1 105209
>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>> java 16908 hadoop mem REG 8,1 134910
>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>> java 16908 hadoop mem REG 8,1 1024681
>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>> java 16908 hadoop mem REG 8,1 15010
>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>> java 16908 hadoop mem REG 8,1 8601
>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>> java 16908 hadoop mem REG 8,1 15345
>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>> java 16908 hadoop mem REG 8,1 132368
>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>> java 16908 hadoop mem REG 8,1 65261
>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>> java 16908 hadoop mem REG 8,1 540852
>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>> java 16908 hadoop mem REG 8,1 391834
>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>> java 16908 hadoop mem REG 8,1 73374
>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>> java 16908 hadoop mem REG 8,1 114342
>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>> java 16908 hadoop mem REG 8,1 11428
>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>> java 16908 hadoop mem REG 8,1 198945
>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>> java 16908 hadoop mem REG 8,1 121070
>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>> java 16908 hadoop mem REG 8,1 163121
>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>> java 16908 hadoop mem REG 8,1 516429
>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>> java 16908 hadoop mem REG 8,1 321806
>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>> java 16908 hadoop mem REG 8,1 76698
>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>> java 16908 hadoop mem REG 8,1 405086
>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>> java 16908 hadoop mem REG 8,1 706710
>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>> java 16908 hadoop mem REG 8,1 706710
>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>> java 16908 hadoop mem REG 8,1 37210
>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>> java 16908 hadoop mem REG 8,1 18684
>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>> java 16908 hadoop mem REG 8,1 37157
>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>> java 16908 hadoop mem REG 8,1 3566844
>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>> java 16908 hadoop mem REG 8,1 180792
>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>> java 16908 hadoop mem REG 8,1 26202
>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>> java 16908 hadoop mem REG 8,1 38015
>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>> java 16908 hadoop mem REG 8,1 279781
>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>> java 16908 hadoop mem REG 8,1 2771746
>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>> java 16908 hadoop mem REG 8,1 12588713
>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>> java 16908 hadoop mem REG 8,1 25700
>>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>>> java 16908 hadoop mem REG 8,1 254076
>>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>>> java 16908 hadoop mem REG 8,1 51669471
>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>> java 16908 hadoop mem REG 8,1 92117
>>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>>> java 16908 hadoop mem REG 8,1 47528
>>>> 1164346 /lib/libnss_files-2.7.so
>>>> java 16908 hadoop mem REG 8,1 43480
>>>> 1164348 /lib/libnss_nis-2.7.so
>>>> java 16908 hadoop mem REG 8,1 35632
>>>> 1164344 /lib/libnss_compat-2.7.so
>>>> java 16908 hadoop mem REG 8,1 55850
>>>> 3555581
>>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>>> java 16908 hadoop mem REG 8,1 93080
>>>> 1164343 /lib/libnsl-2.7.so
>>>> java 16908 hadoop mem REG 8,1 234713
>>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>>> java 16908 hadoop mem REG 8,1 66580
>>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>>> java 16908 hadoop mem REG 8,1 35784
>>>> 1164353 /lib/librt-2.7.so
>>>> java 16908 hadoop mem REG 8,1 526560
>>>> 1164341 /lib/libm-2.7.so
>>>> java 16908 hadoop mem REG 8,1 10987257
>>>> 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>>> java 16908 hadoop mem REG 8,1 1436976
>>>> 1164337 /lib/libc-2.7.so
>>>> java 16908 hadoop mem REG 8,1 14624
>>>> 1164340 /lib/libdl-2.7.so
>>>> java 16908 hadoop mem REG 8,1 130224
>>>> 1164351 /lib/libpthread-2.7.so
>>>> java 16908 hadoop mem REG 8,1 127480
>>>> 1164334 /lib/ld-2.7.so
>>>> java 16908 hadoop mem REG 8,1 618950
>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>> java 16908 hadoop mem REG 8,1 112341
>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>> java 16908 hadoop mem REG 8,1 46725
>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>> java 16908 hadoop mem REG 8,1 32768
>>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>>> java 16908 hadoop mem REG 8,1 48335
>>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>>> java 16908 hadoop mem REG 8,1 41123
>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>> java 16908 hadoop 0u CHR 136,2
>>>> 4 /dev/pts/2 (deleted)
>>>> java 16908 hadoop 1u CHR 136,2
>>>> 4 /dev/pts/2 (deleted)
>>>> java 16908 hadoop 2u CHR 136,2
>>>> 4 /dev/pts/2 (deleted)
>>>> java 16908 hadoop 3r REG 8,1 51669471
>>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>>> java 16908 hadoop 4r REG 8,1 12588713
>>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>>> java 16908 hadoop 5r REG 8,1 2771746
>>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>>> java 16908 hadoop 6r REG 8,1 41123
>>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>>> java 16908 hadoop 7r REG 8,1 46725
>>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>>> java 16908 hadoop 8r REG 8,1 112341
>>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>>> java 16908 hadoop 9r REG 8,1 279781
>>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>>> java 16908 hadoop 10r REG 8,1 38015
>>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>>> java 16908 hadoop 11r REG 8,1 26202
>>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>>> java 16908 hadoop 12r REG 8,1 180792
>>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>>> java 16908 hadoop 13r REG 8,1 3566844
>>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>>> java 16908 hadoop 14r REG 8,1 37157
>>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>>> java 16908 hadoop 15r REG 8,1 18684
>>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>>> java 16908 hadoop 16r REG 8,1 37210
>>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>>> java 16908 hadoop 17r REG 8,1 706710
>>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>>> java 16908 hadoop 18r REG 8,1 706710
>>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>>> java 16908 hadoop 19r REG 8,1 405086
>>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>>> java 16908 hadoop 20r REG 8,1 76698
>>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>>> java 16908 hadoop 21r REG 8,1 321806
>>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>>> java 16908 hadoop 22r REG 8,1 516429
>>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>>> java 16908 hadoop 23r REG 8,1 163121
>>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>>> java 16908 hadoop 24r REG 8,1 121070
>>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>>> java 16908 hadoop 25r REG 8,1 198945
>>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>>> java 16908 hadoop 26r REG 8,1 11428
>>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>>> java 16908 hadoop 27r REG 8,1 114342
>>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>>> java 16908 hadoop 28r REG 8,1 73374
>>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>>> java 16908 hadoop 29r REG 8,1 391834
>>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>>> java 16908 hadoop 30r REG 8,1 540852
>>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>>> java 16908 hadoop 31r REG 8,1 65261
>>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>>> java 16908 hadoop 32r REG 8,1 132368
>>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>>> java 16908 hadoop 33r REG 8,1 15345
>>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>>> java 16908 hadoop 34r REG 8,1 8601
>>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>>> java 16908 hadoop 35r REG 8,1 15010
>>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>>> java 16908 hadoop 36r REG 8,1 1024681
>>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>>> java 16908 hadoop 37r REG 8,1 134910
>>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>>> java 16908 hadoop 38r REG 8,1 105209
>>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>>> java 16908 hadoop 39r REG 8,1 43035
>>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>>> java 16908 hadoop 40r REG 8,1 258337
>>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>>> java 16908 hadoop 41r REG 8,1 46725
>>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>>> java 16908 hadoop 42r REG 8,1 575389
>>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>>> java 16908 hadoop 43r REG 8,1 261809
>>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>>> java 16908 hadoop 44r REG 8,1 38015
>>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>>> java 16908 hadoop 45r REG 8,1 26202
>>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>>> java 16908 hadoop 46r REG 8,1 1923209
>>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>>> java 16908 hadoop 47r REG 8,1 176174
>>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>>> java 16908 hadoop 48r REG 8,1 1154170
>>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>>> java 16908 hadoop 49r REG 8,1 2446767
>>>> 3532072 /usr/lib/hive/lib/derby.jar
>>>> java 16908 hadoop 50r REG 8,1 7092
>>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>>> java 16908 hadoop 51r REG 8,1 12641
>>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>>> java 16908 hadoop 52r REG 8,1 14099
>>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>>> java 16908 hadoop 53r REG 8,1 11412
>>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>>> java 16908 hadoop 54r REG 8,1 1798010
>>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>>> java 16908 hadoop 55r REG 8,1 19349
>>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>>> java 16908 hadoop 56r REG 8,1 31931
>>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>>> java 16908 hadoop 57r REG 8,1 500917
>>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>>> java 16908 hadoop 58r REG 8,1 390114
>>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>>> java 16908 hadoop 59r REG 8,1 95488
>>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>>> java 16908 hadoop 60r REG 8,1 20202
>>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>>> java 16908 hadoop 61r REG 8,1 179508
>>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>>> java 16908 hadoop 62r REG 8,1 87325
>>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>>> java 16908 hadoop 63r REG 8,1 45946
>>>> 3532058 /usr/lib/hive/lib/json.jar
>>>> java 16908 hadoop 64r REG 8,1 121070
>>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>>> java 16908 hadoop 65r REG 8,1 114342
>>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>>> java 16908 hadoop 66r REG 8,1 169666
>>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>>> java 16908 hadoop 67r REG 8,1 391834
>>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>>> java 16908 hadoop 68r REG 8,1 724225
>>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>>> java 16908 hadoop 69r REG 8,1 229928
>>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>>> java 16908 hadoop 70r REG 8,1 392124
>>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>>> java 16908 hadoop 71r CHR 1,8
>>>> 1818 /dev/random
>>>> java 16908 hadoop 72r CHR 1,9
>>>> 1824 /dev/urandom
>>>> java 16908 hadoop 73u IPv6 153142741
>>>> TCP *:webmin (LISTEN)
>>>> java 16908 hadoop 74u sock 0,4
>>>> 153142739 can't identify protocol
>>>> java 16908 hadoop 76w REG 8,1 6867
>>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>>> java 16908 hadoop 77u IPv6 154300915
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 78r REG 8,1 1145980
>>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>>> java 16908 hadoop 79u IPv6 153477399
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 80u IPv6 153594480
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 81w REG 8,1 15491
>>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>>> java 16908 hadoop 82u IPv6 153177590
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 83r REG 8,1 618950
>>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>>> java 16908 hadoop 84u IPv6 153177591
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>>> 153143422 socket
>>>> java 16908 hadoop 86w REG 8,1 62510
>>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>>> java 16908 hadoop 87w REG 8,1 960
>>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>>> java 16908 hadoop 88u IPv6 153219191
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 89u IPv6 153143432
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 90u IPv6 153143434
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 91u IPv6 153219192
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 92u IPv6 153477400
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 93u IPv6 153445420
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 94u IPv6 153385926
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 95u IPv6 153322551
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 96u IPv6 153220498
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 97u IPv6 153220499
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 98u IPv6 153273816
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 99u IPv6 153273819
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 100u IPv6 153293338
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 102u IPv6 153708229
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 103u IPv6 153276086
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 104u IPv6 153276087
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 105u IPv6 153291015
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 106u IPv6 153291016
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 107u IPv6 153293339
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 108u IPv6 153291444
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 109u IPv6 153291445
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 110u IPv6 153294552
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 111u IPv6 153294209
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 112u IPv6 153294210
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 113w REG 8,1 11562
>>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>>> java 16908 hadoop 114u IPv6 153294232
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 115u IPv6 153294233
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 116u IPv6 153294553
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 117u IPv6 153308598
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 118u IPv6 153299396
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 119u IPv6 153299397
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 120u IPv6 153305498
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 121u IPv6 153305501
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 122u IPv6 153322053
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 123u IPv6 153306847
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 124u IPv6 153306848
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 125u IPv6 153308599
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 126u IPv6 153308072
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 127u IPv6 153308073
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 128u IPv6 153322552
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 129u IPv6 153318287
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 130u IPv6 153318288
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 131u IPv6 153322054
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 132u IPv6 153321874
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 133u IPv6 153321875
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 134u IPv6 153708231
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 135u IPv6 153322490
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 136u IPv6 153322491
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 137u IPv6 153330756
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 138u IPv6 153322687
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 139u IPv6 153322688
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 140u IPv6 153325417
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 141u IPv6 153322873
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 142u IPv6 153322874
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 143u IPv6 153325418
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 144u IPv6 153355133
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 145u IPv6 153325537
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 146u IPv6 153325538
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 147u IPv6 153330757
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 148u IPv6 153330673
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 149u IPv6 153330674
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 150u IPv6 153342009
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 151u IPv6 153342010
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 152u IPv6 153343873
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 153u IPv6 153342462
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 154u IPv6 153342463
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 155u IPv6 153343874
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 156u IPv6 153348641
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 157u IPv6 153344781
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 158u IPv6 153344782
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 159u IPv6 153348642
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 160u IPv6 153354839
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 161u IPv6 153353725
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 162u IPv6 153353726
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 163u IPv6 153354840
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 164u IPv6 153363372
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 165u IPv6 153354979
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 166u IPv6 153354980
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 167u IPv6 153355134
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 168u IPv6 153385927
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 169u IPv6 153445421
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 170u IPv6 153405211
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 171u IPv6 153363371
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 172u IPv6 153405212
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 173u IPv6 153373260
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 174u IPv6 153460225
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 175u IPv6 153449798
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 176u IPv6 153360445
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 177u IPv6 153360459
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 178u IPv6 153449799
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 179u IPv6 153373262
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 180u IPv6 153509124
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 181u IPv6 153411087
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 182u IPv6 153411089
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 183u IPv6 153460228
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 184u IPv6 153428120
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 185u IPv6 153428121
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 187u IPv6 153600041
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 188u IPv6 153557843
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 189u IPv6 153462763
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 190u IPv6 153460417
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 191u IPv6 153460418
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 192u IPv6 153462764
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 193u IPv6 153471908
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 194u IPv6 153468521
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 195u IPv6 153468522
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 196u IPv6 153471980
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 197u IPv6 153557844
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 198u IPv6 153476603
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 199u IPv6 153476604
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 200u IPv6 153508848
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 201u IPv6 153507533
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 202u IPv6 153507534
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 203u IPv6 153508419
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 204u IPv6 153508234
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 205u IPv6 153508235
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 206u IPv6 153508422
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 207u IPv6 153509125
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 209u IPv6 153508845
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 210u IPv6 153508846
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 211u IPv6 153572667
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 212u IPv6 153508849
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 213u IPv6 153516692
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 214u IPv6 153516695
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 216u IPv6 153610112
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 217u IPv6 153610114
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 218u IPv6 153557944
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 219u IPv6 153557945
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 220u IPv6 153572668
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 221u IPv6 153585054
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 222u IPv6 153582445
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 223u IPv6 153582447
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 224u IPv6 153585055
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 225u IPv6 153708561
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 226u IPv6 153594476
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 227u IPv6 153638916
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 229u IPv6 153708562
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 230u IPv6 153600134
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 231u IPv6 153638922
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 232u IPv6 153635793
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 233u IPv6 153708619
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 234u IPv6 153703803
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 235u IPv6 153703804
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 236u IPv6 153604398
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 237u IPv6 153604506
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 238u IPv6 153708578
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 241u IPv6 153635795
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 244r FIFO 0,5
>>>> 154318314 pipe
>>>> java 16908 hadoop 245u IPv6 153706355
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 246u IPv6 153706356
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (ESTABLISHED)
>>>> java 16908 hadoop 247w FIFO 0,5
>>>> 154318314 pipe
>>>> java 16908 hadoop 248u IPv6 153708579
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 249w REG 8,1 15230
>>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>>> java 16908 hadoop 250u IPv6 153708593
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 251u IPv6 153708594
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 252u IPv6 153708620
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 253u 0000 0,6 0
>>>> 493 /anon_inode:[eventpoll]
>>>> java 16908 hadoop 255u IPv6 153742520
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 256u IPv6 153742521
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 259u IPv6 153742533
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 260u IPv6 153742534
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 262u IPv6 153742547
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 263u IPv6 153742548
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 265u IPv6 153742562
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> java 16908 hadoop 266u IPv6 153742563
>>>> TCP
>>>>
>>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>>> (CLOSE_WAIT)
>>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>>
>>>>
>>>>
>>>>>
>>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
>>>>> 12345 is the process id of the hive server?
>>>>> We should be able to see the names of the files that are open.
>>>>>
>>>>> Zheng
>>>>>
>>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Nope, no luck so far.
>>>>>>
>>>>>> We have upped the number of file descriptors and are having to restart
>>>>>> hive every week or so :(
>>>>>>
>>>>>> Any other suggestions would be greatly appreciated.
>>>>>>
>>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>>
>>>>>>> Andy Kent wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I can give try and give it a go. I'm not convinced though as we are
>>>>>>>> working with CSV files and don't touch sequence files at all at the
>>>>>>>> moment.
>>>>>>>>
>>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>>> Hive 0.40
>>>>>>>>
>>>>>>>>
>>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>>> where if there were problems opening a file, it would leave a
>>>>>>>>> filehandle
>>>>>>>>> open and never close it.
>>>>>>>>>
>>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some
>>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply
>>>>>>>>> this to
>>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>>
>>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>>> combination.. maybe if you're doing queries and then closing your
>>>>>>>>> thrift
>>>>>>>>> connection before reading all results, Hive doesn't know what to do
>>>>>>>>> and
>>>>>>>>> leaves the connection open? Once the west coast folks wake up,
>>>>>>>>> they might
>>>>>>>>> have a better answer for you than I do.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting
>>>>>>>>>> to
>>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and
>>>>>>>>>> not
>>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for the response.
>>>>>>>>>
>>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby
>>>>>>>>> code
>>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>>
>>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>>> some examples of which ones are leaky. Is this something that I
>>>>>>>>> should post
>>>>>>>>> to Jira or is it a known issue? I can't believe other people
>>>>>>>>> haven't noticed
>>>>>>>>> this?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <SequenceFile.patch>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Yours,
>>>>> Zheng
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
That's the datanode port. So Hdfs. Any trick to see what is opened on
hdfs? like an lsof for hdfs :)
Zheng Shao wrote:
> What's running at victoria.ebuddy.com:50010 ?
>
> Zhemg
>
> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com> wrote:
>
>> I think it's lines like this:
>> java 16908 hadoop 160u IPv6 153354839 TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>>
>> We have the same.
>> Currently no queries running and many of those connections open. After
>> running some queries a few more are added to the list.
>>
>> java 5888 dwh 389u IPv4 1020083623
>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 390u IPv4 1020083626
>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 391u IPv4 1020083637
>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 392u IPv4 1020083641
>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 393u IPv4 1020083668
>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 394u IPv4 1020083675
>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 395u IPv4 1020083678
>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 396u IPv4 1020083690
>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 397u IPv4 1020083693
>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 398u IPv4 1020083703
>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 399u IPv4 1020083706
>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 400u IPv4 1020083713
>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 401u IPv4 1020083716
>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 402u IPv4 1020083728
>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 403u IPv4 1020083731
>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 404u IPv4 1020083740
>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 405u IPv4 1020083743
>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 406u IPv4 1020083753
>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 407u IPv4 1020083756
>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 408u IPv4 1020083769
>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 409u IPv4 1020083772
>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 410u IPv4 1020083781
>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 411u IPv4 1020083784
>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 412u IPv4 1020083792
>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 413u IPv4 1020083795
>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 414u IPv4 1020083808
>> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 415u IPv4 1020083811
>> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 416u IPv4 1020083820
>> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 417u IPv4 1020083823
>> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 418u IPv4 1020083831
>> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 419u IPv4 1020083834
>> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 420u IPv4 1020083846
>> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 421u IPv4 1020083849
>> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 422u IPv4 1020083865
>> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 423u IPv4 1020083862
>> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 424u IPv4 1020083877
>> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 425u IPv4 1020083880
>> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 426u IPv4 1020083888
>> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 427u IPv4 1020083896
>> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 433u IPv4 1020083914
>> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 434u IPv4 1020083916
>> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 435u IPv4 1020083926
>> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 436u IPv4 1020083929
>> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 437u IPv4 1020084012
>> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 438u IPv4 1020083947
>> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 439u IPv4 1020083975
>> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 443u IPv4 1020083950
>> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 444u IPv4 1020083959
>> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 445u IPv4 1020083962
>> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 446u IPv4 1020083977
>> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 447u IPv4 1020083984
>> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 448u IPv4 1020088663
>> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 449u IPv4 1020083991
>> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 450u IPv4 1020088678
>> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 451u IPv4 1020088692
>> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 452u IPv4 1020088694
>> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 453u IPv4 1020092163
>> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 454u IPv4 1020092165
>> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>>
>> Andy Kent wrote:
>>
>>> I have included the output from lsof, Unfortunately though I only
>>> restarted the server a few mins ago and so it doesn't really illustrate the
>>> problem. I'll try to post back with the same command tomorrow so that we can
>>> diff between the two.
>>>
>>> Thanks, Andy.
>>>
>>> java 16908 hadoop cwd DIR 0,17 240
>>> 7037865 /var/run/hadoop-0.20
>>> java 16908 hadoop rtd DIR 8,1 4096
>>> 2 /
>>> java 16908 hadoop txt REG 8,1 50810
>>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>>> java 16908 hadoop mem REG 8,1 127296
>>> 3443156 /usr/lib/liblzo2.so.2.0.0
>>> java 16908 hadoop mem REG 8,1 69472
>>> 3735429
>>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>>> java 16908 hadoop mem REG 8,1 44257
>>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>>> java 16908 hadoop mem REG 8,1 80760
>>> 1164352 /lib/libresolv-2.7.so
>>> java 16908 hadoop mem REG 8,1 22856
>>> 1164345 /lib/libnss_dns-2.7.so
>>> java 16908 hadoop mem REG 8,1 1145980
>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>> java 16908 hadoop mem REG 8,1 38105
>>> 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>>> java 16908 hadoop mem REG 8,1 111901
>>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>>> java 16908 hadoop mem REG 8,1 392124
>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>> java 16908 hadoop mem REG 8,1 229928
>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>> java 16908 hadoop mem REG 8,1 724225
>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>> java 16908 hadoop mem REG 8,1 391834
>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>> java 16908 hadoop mem REG 8,1 169666
>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>> java 16908 hadoop mem REG 8,1 114342
>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>> java 16908 hadoop mem REG 8,1 121070
>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>> java 16908 hadoop mem REG 8,1 45946
>>> 3532058 /usr/lib/hive/lib/json.jar
>>> java 16908 hadoop mem REG 8,1 87325
>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>> java 16908 hadoop mem REG 8,1 179508
>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>> java 16908 hadoop mem REG 8,1 20202
>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>> java 16908 hadoop mem REG 8,1 95488
>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>> java 16908 hadoop mem REG 8,1 390114
>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>> java 16908 hadoop mem REG 8,1 500917
>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>> java 16908 hadoop mem REG 8,1 31931
>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>> java 16908 hadoop mem REG 8,1 19349
>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>> java 16908 hadoop mem REG 8,1 1798010
>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>> java 16908 hadoop mem REG 8,1 11412
>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>> java 16908 hadoop mem REG 8,1 14099
>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>> java 16908 hadoop mem REG 8,1 12641
>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>> java 16908 hadoop mem REG 8,1 7092
>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>> java 16908 hadoop mem REG 8,1 2446767
>>> 3532072 /usr/lib/hive/lib/derby.jar
>>> java 16908 hadoop mem REG 8,1 1154170
>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>> java 16908 hadoop mem REG 8,1 176174
>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>> java 16908 hadoop mem REG 8,1 1923209
>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>> java 16908 hadoop mem REG 8,1 26202
>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>> java 16908 hadoop mem REG 8,1 38015
>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>> java 16908 hadoop mem REG 8,1 261809
>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>> java 16908 hadoop mem REG 8,1 575389
>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>> java 16908 hadoop mem REG 8,1 46725
>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>> java 16908 hadoop mem REG 8,1 258337
>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>> java 16908 hadoop mem REG 8,1 43035
>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>> java 16908 hadoop mem REG 8,1 105209
>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>> java 16908 hadoop mem REG 8,1 134910
>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>> java 16908 hadoop mem REG 8,1 1024681
>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>> java 16908 hadoop mem REG 8,1 15010
>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>> java 16908 hadoop mem REG 8,1 8601
>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>> java 16908 hadoop mem REG 8,1 15345
>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>> java 16908 hadoop mem REG 8,1 132368
>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>> java 16908 hadoop mem REG 8,1 65261
>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>> java 16908 hadoop mem REG 8,1 540852
>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>> java 16908 hadoop mem REG 8,1 391834
>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>> java 16908 hadoop mem REG 8,1 73374
>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>> java 16908 hadoop mem REG 8,1 114342
>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>> java 16908 hadoop mem REG 8,1 11428
>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>> java 16908 hadoop mem REG 8,1 198945
>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>> java 16908 hadoop mem REG 8,1 121070
>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>> java 16908 hadoop mem REG 8,1 163121
>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>> java 16908 hadoop mem REG 8,1 516429
>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>> java 16908 hadoop mem REG 8,1 321806
>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>> java 16908 hadoop mem REG 8,1 76698
>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>> java 16908 hadoop mem REG 8,1 405086
>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>> java 16908 hadoop mem REG 8,1 706710
>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>> java 16908 hadoop mem REG 8,1 706710
>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>> java 16908 hadoop mem REG 8,1 37210
>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>> java 16908 hadoop mem REG 8,1 18684
>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>> java 16908 hadoop mem REG 8,1 37157
>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>> java 16908 hadoop mem REG 8,1 3566844
>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>> java 16908 hadoop mem REG 8,1 180792
>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>> java 16908 hadoop mem REG 8,1 26202
>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>> java 16908 hadoop mem REG 8,1 38015
>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>> java 16908 hadoop mem REG 8,1 279781
>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>> java 16908 hadoop mem REG 8,1 2771746
>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>> java 16908 hadoop mem REG 8,1 12588713
>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>> java 16908 hadoop mem REG 8,1 25700
>>> 3441590 /usr/lib/gconv/gconv-modules.cache
>>> java 16908 hadoop mem REG 8,1 254076
>>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>>> java 16908 hadoop mem REG 8,1 51669471
>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>> java 16908 hadoop mem REG 8,1 92117
>>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>>> java 16908 hadoop mem REG 8,1 47528
>>> 1164346 /lib/libnss_files-2.7.so
>>> java 16908 hadoop mem REG 8,1 43480
>>> 1164348 /lib/libnss_nis-2.7.so
>>> java 16908 hadoop mem REG 8,1 35632
>>> 1164344 /lib/libnss_compat-2.7.so
>>> java 16908 hadoop mem REG 8,1 55850
>>> 3555581
>>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>>> java 16908 hadoop mem REG 8,1 93080
>>> 1164343 /lib/libnsl-2.7.so
>>> java 16908 hadoop mem REG 8,1 234713
>>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>>> java 16908 hadoop mem REG 8,1 66580
>>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>>> java 16908 hadoop mem REG 8,1 35784
>>> 1164353 /lib/librt-2.7.so
>>> java 16908 hadoop mem REG 8,1 526560
>>> 1164341 /lib/libm-2.7.so
>>> java 16908 hadoop mem REG 8,1 10987257
>>> 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>>> java 16908 hadoop mem REG 8,1 1436976
>>> 1164337 /lib/libc-2.7.so
>>> java 16908 hadoop mem REG 8,1 14624
>>> 1164340 /lib/libdl-2.7.so
>>> java 16908 hadoop mem REG 8,1 130224
>>> 1164351 /lib/libpthread-2.7.so
>>> java 16908 hadoop mem REG 8,1 127480
>>> 1164334 /lib/ld-2.7.so
>>> java 16908 hadoop mem REG 8,1 618950
>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>> java 16908 hadoop mem REG 8,1 112341
>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>> java 16908 hadoop mem REG 8,1 46725
>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>> java 16908 hadoop mem REG 8,1 32768
>>> 3989505 /tmp/hsperfdata_hadoop/16908
>>> java 16908 hadoop mem REG 8,1 48335
>>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>>> java 16908 hadoop mem REG 8,1 41123
>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>> java 16908 hadoop 0u CHR 136,2
>>> 4 /dev/pts/2 (deleted)
>>> java 16908 hadoop 1u CHR 136,2
>>> 4 /dev/pts/2 (deleted)
>>> java 16908 hadoop 2u CHR 136,2
>>> 4 /dev/pts/2 (deleted)
>>> java 16908 hadoop 3r REG 8,1 51669471
>>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>>> java 16908 hadoop 4r REG 8,1 12588713
>>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>>> java 16908 hadoop 5r REG 8,1 2771746
>>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>>> java 16908 hadoop 6r REG 8,1 41123
>>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>>> java 16908 hadoop 7r REG 8,1 46725
>>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>>> java 16908 hadoop 8r REG 8,1 112341
>>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>>> java 16908 hadoop 9r REG 8,1 279781
>>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>>> java 16908 hadoop 10r REG 8,1 38015
>>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>>> java 16908 hadoop 11r REG 8,1 26202
>>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>>> java 16908 hadoop 12r REG 8,1 180792
>>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>>> java 16908 hadoop 13r REG 8,1 3566844
>>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>>> java 16908 hadoop 14r REG 8,1 37157
>>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>>> java 16908 hadoop 15r REG 8,1 18684
>>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>>> java 16908 hadoop 16r REG 8,1 37210
>>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>>> java 16908 hadoop 17r REG 8,1 706710
>>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>>> java 16908 hadoop 18r REG 8,1 706710
>>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>>> java 16908 hadoop 19r REG 8,1 405086
>>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>>> java 16908 hadoop 20r REG 8,1 76698
>>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>>> java 16908 hadoop 21r REG 8,1 321806
>>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>>> java 16908 hadoop 22r REG 8,1 516429
>>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>>> java 16908 hadoop 23r REG 8,1 163121
>>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>>> java 16908 hadoop 24r REG 8,1 121070
>>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>>> java 16908 hadoop 25r REG 8,1 198945
>>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>>> java 16908 hadoop 26r REG 8,1 11428
>>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>>> java 16908 hadoop 27r REG 8,1 114342
>>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>>> java 16908 hadoop 28r REG 8,1 73374
>>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>>> java 16908 hadoop 29r REG 8,1 391834
>>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>>> java 16908 hadoop 30r REG 8,1 540852
>>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>>> java 16908 hadoop 31r REG 8,1 65261
>>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>>> java 16908 hadoop 32r REG 8,1 132368
>>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>>> java 16908 hadoop 33r REG 8,1 15345
>>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>>> java 16908 hadoop 34r REG 8,1 8601
>>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>>> java 16908 hadoop 35r REG 8,1 15010
>>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>>> java 16908 hadoop 36r REG 8,1 1024681
>>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>>> java 16908 hadoop 37r REG 8,1 134910
>>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>>> java 16908 hadoop 38r REG 8,1 105209
>>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>>> java 16908 hadoop 39r REG 8,1 43035
>>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>>> java 16908 hadoop 40r REG 8,1 258337
>>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>>> java 16908 hadoop 41r REG 8,1 46725
>>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>>> java 16908 hadoop 42r REG 8,1 575389
>>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>>> java 16908 hadoop 43r REG 8,1 261809
>>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>>> java 16908 hadoop 44r REG 8,1 38015
>>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>>> java 16908 hadoop 45r REG 8,1 26202
>>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>>> java 16908 hadoop 46r REG 8,1 1923209
>>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>>> java 16908 hadoop 47r REG 8,1 176174
>>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>>> java 16908 hadoop 48r REG 8,1 1154170
>>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>>> java 16908 hadoop 49r REG 8,1 2446767
>>> 3532072 /usr/lib/hive/lib/derby.jar
>>> java 16908 hadoop 50r REG 8,1 7092
>>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>>> java 16908 hadoop 51r REG 8,1 12641
>>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>>> java 16908 hadoop 52r REG 8,1 14099
>>> 3532074 /usr/lib/hive/lib/hive_common.jar
>>> java 16908 hadoop 53r REG 8,1 11412
>>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>>> java 16908 hadoop 54r REG 8,1 1798010
>>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>>> java 16908 hadoop 55r REG 8,1 19349
>>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>>> java 16908 hadoop 56r REG 8,1 31931
>>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>>> java 16908 hadoop 57r REG 8,1 500917
>>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>>> java 16908 hadoop 58r REG 8,1 390114
>>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>>> java 16908 hadoop 59r REG 8,1 95488
>>> 3532068 /usr/lib/hive/lib/hive_service.jar
>>> java 16908 hadoop 60r REG 8,1 20202
>>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>>> java 16908 hadoop 61r REG 8,1 179508
>>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>>> java 16908 hadoop 62r REG 8,1 87325
>>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>>> java 16908 hadoop 63r REG 8,1 45946
>>> 3532058 /usr/lib/hive/lib/json.jar
>>> java 16908 hadoop 64r REG 8,1 121070
>>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>>> java 16908 hadoop 65r REG 8,1 114342
>>> 3532078 /usr/lib/hive/lib/libfb303.jar
>>> java 16908 hadoop 66r REG 8,1 169666
>>> 3532090 /usr/lib/hive/lib/libthrift.jar
>>> java 16908 hadoop 67r REG 8,1 391834
>>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>>> java 16908 hadoop 68r REG 8,1 724225
>>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>>> java 16908 hadoop 69r REG 8,1 229928
>>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>>> java 16908 hadoop 70r REG 8,1 392124
>>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>>> java 16908 hadoop 71r CHR 1,8
>>> 1818 /dev/random
>>> java 16908 hadoop 72r CHR 1,9
>>> 1824 /dev/urandom
>>> java 16908 hadoop 73u IPv6 153142741
>>> TCP *:webmin (LISTEN)
>>> java 16908 hadoop 74u sock 0,4
>>> 153142739 can't identify protocol
>>> java 16908 hadoop 76w REG 8,1 6867
>>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>>> java 16908 hadoop 77u IPv6 154300915
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 78r REG 8,1 1145980
>>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>>> java 16908 hadoop 79u IPv6 153477399
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 80u IPv6 153594480
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 81w REG 8,1 15491
>>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>>> java 16908 hadoop 82u IPv6 153177590
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 83r REG 8,1 618950
>>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>>> java 16908 hadoop 84u IPv6 153177591
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 85u unix 0xffff8101b5090000
>>> 153143422 socket
>>> java 16908 hadoop 86w REG 8,1 62510
>>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>>> java 16908 hadoop 87w REG 8,1 960
>>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>>> java 16908 hadoop 88u IPv6 153219191
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 89u IPv6 153143432
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 90u IPv6 153143434
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 91u IPv6 153219192
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 92u IPv6 153477400
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 93u IPv6 153445420
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 94u IPv6 153385926
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 95u IPv6 153322551
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 96u IPv6 153220498
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 97u IPv6 153220499
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 98u IPv6 153273816
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 99u IPv6 153273819
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 100u IPv6 153293338
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 102u IPv6 153708229
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 103u IPv6 153276086
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 104u IPv6 153276087
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 105u IPv6 153291015
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 106u IPv6 153291016
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 107u IPv6 153293339
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 108u IPv6 153291444
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 109u IPv6 153291445
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 110u IPv6 153294552
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 111u IPv6 153294209
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 112u IPv6 153294210
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 113w REG 8,1 11562
>>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>>> java 16908 hadoop 114u IPv6 153294232
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 115u IPv6 153294233
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 116u IPv6 153294553
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 117u IPv6 153308598
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 118u IPv6 153299396
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 119u IPv6 153299397
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 120u IPv6 153305498
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 121u IPv6 153305501
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 122u IPv6 153322053
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 123u IPv6 153306847
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 124u IPv6 153306848
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 125u IPv6 153308599
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 126u IPv6 153308072
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 127u IPv6 153308073
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 128u IPv6 153322552
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 129u IPv6 153318287
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 130u IPv6 153318288
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 131u IPv6 153322054
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 132u IPv6 153321874
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 133u IPv6 153321875
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 134u IPv6 153708231
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 135u IPv6 153322490
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 136u IPv6 153322491
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 137u IPv6 153330756
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 138u IPv6 153322687
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 139u IPv6 153322688
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 140u IPv6 153325417
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 141u IPv6 153322873
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 142u IPv6 153322874
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 143u IPv6 153325418
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 144u IPv6 153355133
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 145u IPv6 153325537
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 146u IPv6 153325538
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 147u IPv6 153330757
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 148u IPv6 153330673
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 149u IPv6 153330674
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 150u IPv6 153342009
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 151u IPv6 153342010
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 152u IPv6 153343873
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 153u IPv6 153342462
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 154u IPv6 153342463
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 155u IPv6 153343874
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 156u IPv6 153348641
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 157u IPv6 153344781
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 158u IPv6 153344782
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 159u IPv6 153348642
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 160u IPv6 153354839
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 161u IPv6 153353725
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 162u IPv6 153353726
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 163u IPv6 153354840
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 164u IPv6 153363372
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 165u IPv6 153354979
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 166u IPv6 153354980
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 167u IPv6 153355134
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 168u IPv6 153385927
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 169u IPv6 153445421
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 170u IPv6 153405211
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 171u IPv6 153363371
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 172u IPv6 153405212
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 173u IPv6 153373260
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 174u IPv6 153460225
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 175u IPv6 153449798
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 176u IPv6 153360445
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 177u IPv6 153360459
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 178u IPv6 153449799
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 179u IPv6 153373262
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 180u IPv6 153509124
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 181u IPv6 153411087
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 182u IPv6 153411089
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 183u IPv6 153460228
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 184u IPv6 153428120
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 185u IPv6 153428121
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 187u IPv6 153600041
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 188u IPv6 153557843
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 189u IPv6 153462763
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 190u IPv6 153460417
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 191u IPv6 153460418
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 192u IPv6 153462764
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 193u IPv6 153471908
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 194u IPv6 153468521
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 195u IPv6 153468522
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 196u IPv6 153471980
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 197u IPv6 153557844
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 198u IPv6 153476603
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 199u IPv6 153476604
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 200u IPv6 153508848
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 201u IPv6 153507533
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 202u IPv6 153507534
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 203u IPv6 153508419
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 204u IPv6 153508234
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 205u IPv6 153508235
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 206u IPv6 153508422
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 207u IPv6 153509125
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 209u IPv6 153508845
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 210u IPv6 153508846
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 211u IPv6 153572667
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 212u IPv6 153508849
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 213u IPv6 153516692
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 214u IPv6 153516695
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 216u IPv6 153610112
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 217u IPv6 153610114
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 218u IPv6 153557944
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 219u IPv6 153557945
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 220u IPv6 153572668
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 221u IPv6 153585054
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 222u IPv6 153582445
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 223u IPv6 153582447
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 224u IPv6 153585055
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 225u IPv6 153708561
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 226u IPv6 153594476
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 227u IPv6 153638916
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 229u IPv6 153708562
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 230u IPv6 153600134
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 231u IPv6 153638922
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 232u IPv6 153635793
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 233u IPv6 153708619
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 234u IPv6 153703803
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 235u IPv6 153703804
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 236u IPv6 153604398
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 237u IPv6 153604506
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 238u IPv6 153708578
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 241u IPv6 153635795
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 244r FIFO 0,5
>>> 154318314 pipe
>>> java 16908 hadoop 245u IPv6 153706355
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 246u IPv6 153706356
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>> java 16908 hadoop 247w FIFO 0,5
>>> 154318314 pipe
>>> java 16908 hadoop 248u IPv6 153708579
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 249w REG 8,1 15230
>>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>>> java 16908 hadoop 250u IPv6 153708593
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 251u IPv6 153708594
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 252u IPv6 153708620
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 253u 0000 0,6 0
>>> 493 /anon_inode:[eventpoll]
>>> java 16908 hadoop 255u IPv6 153742520
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 256u IPv6 153742521
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 259u IPv6 153742533
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 260u IPv6 153742534
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 262u IPv6 153742547
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 263u IPv6 153742548
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 265u IPv6 153742562
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> java 16908 hadoop 266u IPv6 153742563
>>> TCP
>>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>>> (CLOSE_WAIT)
>>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>>
>>>
>>>
>>>> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
>>>> 12345 is the process id of the hive server?
>>>> We should be able to see the names of the files that are open.
>>>>
>>>> Zheng
>>>>
>>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>>> wrote:
>>>>
>>>>
>>>>> Nope, no luck so far.
>>>>>
>>>>> We have upped the number of file descriptors and are having to restart
>>>>> hive every week or so :(
>>>>>
>>>>> Any other suggestions would be greatly appreciated.
>>>>>
>>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>>
>>>>>> Andy Kent wrote:
>>>>>>
>>>>>>
>>>>>>> I can give try and give it a go. I'm not convinced though as we are
>>>>>>> working with CSV files and don't touch sequence files at all at the moment.
>>>>>>>
>>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>>> Hive 0.40
>>>>>>>
>>>>>>>
>>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>>> where if there were problems opening a file, it would leave a filehandle
>>>>>>>> open and never close it.
>>>>>>>>
>>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some
>>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply this to
>>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>>
>>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>>> combination.. maybe if you're doing queries and then closing your thrift
>>>>>>>> connection before reading all results, Hive doesn't know what to do and
>>>>>>>> leaves the connection open? Once the west coast folks wake up, they might
>>>>>>>> have a better answer for you than I do.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting to
>>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and not
>>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks for the response.
>>>>>>>>
>>>>>>>> That's what I was suspecting. I have triple checked and our Ruby code
>>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>>
>>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>>> some examples of which ones are leaky. Is this something that I should post
>>>>>>>> to Jira or is it a known issue? I can't believe other people haven't noticed
>>>>>>>> this?
>>>>>>>>
>>>>>>>>
>>>>>>>> <SequenceFile.patch>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>>
>>>>
>>>
>>
>
>
>
> --
> Yours,
> Zheng
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
What's running at victoria.ebuddy.com:50010 ?
Zhemg
On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <bs...@ebuddy.com> wrote:
> I think it's lines like this:
> java 16908 hadoop 160u IPv6 153354839 TCP
> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
> (ESTABLISHED)
>
> We have the same.
> Currently no queries running and many of those connections open. After
> running some queries a few more are added to the list.
>
> java 5888 dwh 389u IPv4 1020083623
> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 390u IPv4 1020083626
> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 391u IPv4 1020083637
> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 392u IPv4 1020083641
> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 393u IPv4 1020083668
> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 394u IPv4 1020083675
> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 395u IPv4 1020083678
> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 396u IPv4 1020083690
> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 397u IPv4 1020083693
> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 398u IPv4 1020083703
> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 399u IPv4 1020083706
> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 400u IPv4 1020083713
> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 401u IPv4 1020083716
> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 402u IPv4 1020083728
> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 403u IPv4 1020083731
> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 404u IPv4 1020083740
> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 405u IPv4 1020083743
> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 406u IPv4 1020083753
> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 407u IPv4 1020083756
> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 408u IPv4 1020083769
> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 409u IPv4 1020083772
> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 410u IPv4 1020083781
> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 411u IPv4 1020083784
> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 412u IPv4 1020083792
> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 413u IPv4 1020083795
> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 414u IPv4 1020083808
> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 415u IPv4 1020083811
> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 416u IPv4 1020083820
> TCP victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 417u IPv4 1020083823
> TCP victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 418u IPv4 1020083831
> TCP victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 419u IPv4 1020083834
> TCP victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 420u IPv4 1020083846
> TCP victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 421u IPv4 1020083849
> TCP victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 422u IPv4 1020083865
> TCP victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 423u IPv4 1020083862
> TCP victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 424u IPv4 1020083877
> TCP victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 425u IPv4 1020083880
> TCP victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 426u IPv4 1020083888
> TCP victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 427u IPv4 1020083896
> TCP victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 433u IPv4 1020083914
> TCP victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 434u IPv4 1020083916
> TCP victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 435u IPv4 1020083926
> TCP victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 436u IPv4 1020083929
> TCP victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 437u IPv4 1020084012
> TCP victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 438u IPv4 1020083947
> TCP victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 439u IPv4 1020083975
> TCP victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 443u IPv4 1020083950
> TCP victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 444u IPv4 1020083959
> TCP victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 445u IPv4 1020083962
> TCP victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 446u IPv4 1020083977
> TCP victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 447u IPv4 1020083984
> TCP victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 448u IPv4 1020088663
> TCP victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 449u IPv4 1020083991
> TCP victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 450u IPv4 1020088678
> TCP victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 451u IPv4 1020088692
> TCP victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 452u IPv4 1020088694
> TCP victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 453u IPv4 1020092163
> TCP victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 454u IPv4 1020092165
> TCP victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
>
> Andy Kent wrote:
>>
>> I have included the output from lsof, Unfortunately though I only
>> restarted the server a few mins ago and so it doesn't really illustrate the
>> problem. I'll try to post back with the same command tomorrow so that we can
>> diff between the two.
>>
>> Thanks, Andy.
>>
>> java 16908 hadoop cwd DIR 0,17 240
>> 7037865 /var/run/hadoop-0.20
>> java 16908 hadoop rtd DIR 8,1 4096
>> 2 /
>> java 16908 hadoop txt REG 8,1 50810
>> 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
>> java 16908 hadoop mem REG 8,1 127296
>> 3443156 /usr/lib/liblzo2.so.2.0.0
>> java 16908 hadoop mem REG 8,1 69472
>> 3735429
>> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
>> java 16908 hadoop mem REG 8,1 44257
>> 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
>> java 16908 hadoop mem REG 8,1 80760
>> 1164352 /lib/libresolv-2.7.so
>> java 16908 hadoop mem REG 8,1 22856
>> 1164345 /lib/libnss_dns-2.7.so
>> java 16908 hadoop mem REG 8,1 1145980
>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>> java 16908 hadoop mem REG 8,1 38105
>> 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
>> java 16908 hadoop mem REG 8,1 111901
>> 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
>> java 16908 hadoop mem REG 8,1 392124
>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>> java 16908 hadoop mem REG 8,1 229928
>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>> java 16908 hadoop mem REG 8,1 724225
>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>> java 16908 hadoop mem REG 8,1 391834
>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>> java 16908 hadoop mem REG 8,1 169666
>> 3532090 /usr/lib/hive/lib/libthrift.jar
>> java 16908 hadoop mem REG 8,1 114342
>> 3532078 /usr/lib/hive/lib/libfb303.jar
>> java 16908 hadoop mem REG 8,1 121070
>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>> java 16908 hadoop mem REG 8,1 45946
>> 3532058 /usr/lib/hive/lib/json.jar
>> java 16908 hadoop mem REG 8,1 87325
>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>> java 16908 hadoop mem REG 8,1 179508
>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>> java 16908 hadoop mem REG 8,1 20202
>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>> java 16908 hadoop mem REG 8,1 95488
>> 3532068 /usr/lib/hive/lib/hive_service.jar
>> java 16908 hadoop mem REG 8,1 390114
>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>> java 16908 hadoop mem REG 8,1 500917
>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>> java 16908 hadoop mem REG 8,1 31931
>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>> java 16908 hadoop mem REG 8,1 19349
>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>> java 16908 hadoop mem REG 8,1 1798010
>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>> java 16908 hadoop mem REG 8,1 11412
>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>> java 16908 hadoop mem REG 8,1 14099
>> 3532074 /usr/lib/hive/lib/hive_common.jar
>> java 16908 hadoop mem REG 8,1 12641
>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>> java 16908 hadoop mem REG 8,1 7092
>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>> java 16908 hadoop mem REG 8,1 2446767
>> 3532072 /usr/lib/hive/lib/derby.jar
>> java 16908 hadoop mem REG 8,1 1154170
>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>> java 16908 hadoop mem REG 8,1 176174
>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>> java 16908 hadoop mem REG 8,1 1923209
>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>> java 16908 hadoop mem REG 8,1 26202
>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>> java 16908 hadoop mem REG 8,1 38015
>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>> java 16908 hadoop mem REG 8,1 261809
>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>> java 16908 hadoop mem REG 8,1 575389
>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>> java 16908 hadoop mem REG 8,1 46725
>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>> java 16908 hadoop mem REG 8,1 258337
>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>> java 16908 hadoop mem REG 8,1 43035
>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>> java 16908 hadoop mem REG 8,1 105209
>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>> java 16908 hadoop mem REG 8,1 134910
>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>> java 16908 hadoop mem REG 8,1 1024681
>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>> java 16908 hadoop mem REG 8,1 15010
>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>> java 16908 hadoop mem REG 8,1 8601
>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>> java 16908 hadoop mem REG 8,1 15345
>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>> java 16908 hadoop mem REG 8,1 132368
>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>> java 16908 hadoop mem REG 8,1 65261
>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>> java 16908 hadoop mem REG 8,1 540852
>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>> java 16908 hadoop mem REG 8,1 391834
>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>> java 16908 hadoop mem REG 8,1 73374
>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>> java 16908 hadoop mem REG 8,1 114342
>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>> java 16908 hadoop mem REG 8,1 11428
>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>> java 16908 hadoop mem REG 8,1 198945
>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>> java 16908 hadoop mem REG 8,1 121070
>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>> java 16908 hadoop mem REG 8,1 163121
>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>> java 16908 hadoop mem REG 8,1 516429
>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>> java 16908 hadoop mem REG 8,1 321806
>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>> java 16908 hadoop mem REG 8,1 76698
>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>> java 16908 hadoop mem REG 8,1 405086
>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>> java 16908 hadoop mem REG 8,1 706710
>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>> java 16908 hadoop mem REG 8,1 706710
>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>> java 16908 hadoop mem REG 8,1 37210
>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>> java 16908 hadoop mem REG 8,1 18684
>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>> java 16908 hadoop mem REG 8,1 37157
>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>> java 16908 hadoop mem REG 8,1 3566844
>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>> java 16908 hadoop mem REG 8,1 180792
>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>> java 16908 hadoop mem REG 8,1 26202
>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>> java 16908 hadoop mem REG 8,1 38015
>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>> java 16908 hadoop mem REG 8,1 279781
>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>> java 16908 hadoop mem REG 8,1 2771746
>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>> java 16908 hadoop mem REG 8,1 12588713
>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>> java 16908 hadoop mem REG 8,1 25700
>> 3441590 /usr/lib/gconv/gconv-modules.cache
>> java 16908 hadoop mem REG 8,1 254076
>> 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
>> java 16908 hadoop mem REG 8,1 51669471
>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>> java 16908 hadoop mem REG 8,1 92117
>> 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
>> java 16908 hadoop mem REG 8,1 47528
>> 1164346 /lib/libnss_files-2.7.so
>> java 16908 hadoop mem REG 8,1 43480
>> 1164348 /lib/libnss_nis-2.7.so
>> java 16908 hadoop mem REG 8,1 35632
>> 1164344 /lib/libnss_compat-2.7.so
>> java 16908 hadoop mem REG 8,1 55850
>> 3555581
>> /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
>> java 16908 hadoop mem REG 8,1 93080
>> 1164343 /lib/libnsl-2.7.so
>> java 16908 hadoop mem REG 8,1 234713
>> 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
>> java 16908 hadoop mem REG 8,1 66580
>> 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
>> java 16908 hadoop mem REG 8,1 35784
>> 1164353 /lib/librt-2.7.so
>> java 16908 hadoop mem REG 8,1 526560
>> 1164341 /lib/libm-2.7.so
>> java 16908 hadoop mem REG 8,1 10987257
>> 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
>> java 16908 hadoop mem REG 8,1 1436976
>> 1164337 /lib/libc-2.7.so
>> java 16908 hadoop mem REG 8,1 14624
>> 1164340 /lib/libdl-2.7.so
>> java 16908 hadoop mem REG 8,1 130224
>> 1164351 /lib/libpthread-2.7.so
>> java 16908 hadoop mem REG 8,1 127480
>> 1164334 /lib/ld-2.7.so
>> java 16908 hadoop mem REG 8,1 618950
>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>> java 16908 hadoop mem REG 8,1 112341
>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>> java 16908 hadoop mem REG 8,1 46725
>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>> java 16908 hadoop mem REG 8,1 32768
>> 3989505 /tmp/hsperfdata_hadoop/16908
>> java 16908 hadoop mem REG 8,1 48335
>> 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
>> java 16908 hadoop mem REG 8,1 41123
>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>> java 16908 hadoop 0u CHR 136,2
>> 4 /dev/pts/2 (deleted)
>> java 16908 hadoop 1u CHR 136,2
>> 4 /dev/pts/2 (deleted)
>> java 16908 hadoop 2u CHR 136,2
>> 4 /dev/pts/2 (deleted)
>> java 16908 hadoop 3r REG 8,1 51669471
>> 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
>> java 16908 hadoop 4r REG 8,1 12588713
>> 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
>> java 16908 hadoop 5r REG 8,1 2771746
>> 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
>> java 16908 hadoop 6r REG 8,1 41123
>> 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
>> java 16908 hadoop 7r REG 8,1 46725
>> 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
>> java 16908 hadoop 8r REG 8,1 112341
>> 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
>> java 16908 hadoop 9r REG 8,1 279781
>> 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
>> java 16908 hadoop 10r REG 8,1 38015
>> 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
>> java 16908 hadoop 11r REG 8,1 26202
>> 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
>> java 16908 hadoop 12r REG 8,1 180792
>> 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
>> java 16908 hadoop 13r REG 8,1 3566844
>> 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
>> java 16908 hadoop 14r REG 8,1 37157
>> 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
>> java 16908 hadoop 15r REG 8,1 18684
>> 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
>> java 16908 hadoop 16r REG 8,1 37210
>> 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
>> java 16908 hadoop 17r REG 8,1 706710
>> 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
>> java 16908 hadoop 18r REG 8,1 706710
>> 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
>> java 16908 hadoop 19r REG 8,1 405086
>> 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
>> java 16908 hadoop 20r REG 8,1 76698
>> 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
>> java 16908 hadoop 21r REG 8,1 321806
>> 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
>> java 16908 hadoop 22r REG 8,1 516429
>> 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
>> java 16908 hadoop 23r REG 8,1 163121
>> 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
>> java 16908 hadoop 24r REG 8,1 121070
>> 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
>> java 16908 hadoop 25r REG 8,1 198945
>> 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
>> java 16908 hadoop 26r REG 8,1 11428
>> 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
>> java 16908 hadoop 27r REG 8,1 114342
>> 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
>> java 16908 hadoop 28r REG 8,1 73374
>> 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
>> java 16908 hadoop 29r REG 8,1 391834
>> 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
>> java 16908 hadoop 30r REG 8,1 540852
>> 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
>> java 16908 hadoop 31r REG 8,1 65261
>> 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
>> java 16908 hadoop 32r REG 8,1 132368
>> 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
>> java 16908 hadoop 33r REG 8,1 15345
>> 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
>> java 16908 hadoop 34r REG 8,1 8601
>> 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
>> java 16908 hadoop 35r REG 8,1 15010
>> 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
>> java 16908 hadoop 36r REG 8,1 1024681
>> 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
>> java 16908 hadoop 37r REG 8,1 134910
>> 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
>> java 16908 hadoop 38r REG 8,1 105209
>> 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
>> java 16908 hadoop 39r REG 8,1 43035
>> 3532057 /usr/lib/hive/lib/asm-3.1.jar
>> java 16908 hadoop 40r REG 8,1 258337
>> 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
>> java 16908 hadoop 41r REG 8,1 46725
>> 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
>> java 16908 hadoop 42r REG 8,1 575389
>> 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
>> java 16908 hadoop 43r REG 8,1 261809
>> 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
>> java 16908 hadoop 44r REG 8,1 38015
>> 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
>> java 16908 hadoop 45r REG 8,1 26202
>> 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
>> java 16908 hadoop 46r REG 8,1 1923209
>> 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
>> java 16908 hadoop 47r REG 8,1 176174
>> 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
>> java 16908 hadoop 48r REG 8,1 1154170
>> 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
>> java 16908 hadoop 49r REG 8,1 2446767
>> 3532072 /usr/lib/hive/lib/derby.jar
>> java 16908 hadoop 50r REG 8,1 7092
>> 3532067 /usr/lib/hive/lib/hive_anttasks.jar
>> java 16908 hadoop 51r REG 8,1 12641
>> 3532064 /usr/lib/hive/lib/hive_cli.jar
>> java 16908 hadoop 52r REG 8,1 14099
>> 3532074 /usr/lib/hive/lib/hive_common.jar
>> java 16908 hadoop 53r REG 8,1 11412
>> 3532091 /usr/lib/hive/lib/hive_contrib.jar
>> java 16908 hadoop 54r REG 8,1 1798010
>> 3532071 /usr/lib/hive/lib/hive_exec.jar
>> java 16908 hadoop 55r REG 8,1 19349
>> 3532081 /usr/lib/hive/lib/hive_hwi.jar
>> java 16908 hadoop 56r REG 8,1 31931
>> 3532070 /usr/lib/hive/lib/hive_jdbc.jar
>> java 16908 hadoop 57r REG 8,1 500917
>> 3532063 /usr/lib/hive/lib/hive_metastore.jar
>> java 16908 hadoop 58r REG 8,1 390114
>> 3532076 /usr/lib/hive/lib/hive_serde.jar
>> java 16908 hadoop 59r REG 8,1 95488
>> 3532068 /usr/lib/hive/lib/hive_service.jar
>> java 16908 hadoop 60r REG 8,1 20202
>> 3532073 /usr/lib/hive/lib/hive_shims.jar
>> java 16908 hadoop 61r REG 8,1 179508
>> 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
>> java 16908 hadoop 62r REG 8,1 87325
>> 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
>> java 16908 hadoop 63r REG 8,1 45946
>> 3532058 /usr/lib/hive/lib/json.jar
>> java 16908 hadoop 64r REG 8,1 121070
>> 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
>> java 16908 hadoop 65r REG 8,1 114342
>> 3532078 /usr/lib/hive/lib/libfb303.jar
>> java 16908 hadoop 66r REG 8,1 169666
>> 3532090 /usr/lib/hive/lib/libthrift.jar
>> java 16908 hadoop 67r REG 8,1 391834
>> 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
>> java 16908 hadoop 68r REG 8,1 724225
>> 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
>> java 16908 hadoop 69r REG 8,1 229928
>> 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
>> java 16908 hadoop 70r REG 8,1 392124
>> 3532065 /usr/lib/hive/lib/velocity-1.5.jar
>> java 16908 hadoop 71r CHR 1,8
>> 1818 /dev/random
>> java 16908 hadoop 72r CHR 1,9
>> 1824 /dev/urandom
>> java 16908 hadoop 73u IPv6 153142741
>> TCP *:webmin (LISTEN)
>> java 16908 hadoop 74u sock 0,4
>> 153142739 can't identify protocol
>> java 16908 hadoop 76w REG 8,1 6867
>> 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
>> java 16908 hadoop 77u IPv6 154300915
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 78r REG 8,1 1145980
>> 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
>> java 16908 hadoop 79u IPv6 153477399
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 80u IPv6 153594480
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 81w REG 8,1 15491
>> 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
>> java 16908 hadoop 82u IPv6 153177590
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 83r REG 8,1 618950
>> 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
>> java 16908 hadoop 84u IPv6 153177591
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 85u unix 0xffff8101b5090000
>> 153143422 socket
>> java 16908 hadoop 86w REG 8,1 62510
>> 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
>> java 16908 hadoop 87w REG 8,1 960
>> 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
>> java 16908 hadoop 88u IPv6 153219191
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 89u IPv6 153143432
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 90u IPv6 153143434
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 91u IPv6 153219192
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 92u IPv6 153477400
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 93u IPv6 153445420
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 94u IPv6 153385926
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 95u IPv6 153322551
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 96u IPv6 153220498
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 97u IPv6 153220499
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 98u IPv6 153273816
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 99u IPv6 153273819
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 100u IPv6 153293338
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 102u IPv6 153708229
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 103u IPv6 153276086
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 104u IPv6 153276087
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 105u IPv6 153291015
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 106u IPv6 153291016
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 107u IPv6 153293339
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 108u IPv6 153291444
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 109u IPv6 153291445
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 110u IPv6 153294552
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 111u IPv6 153294209
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 112u IPv6 153294210
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 113w REG 8,1 11562
>> 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
>> java 16908 hadoop 114u IPv6 153294232
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 115u IPv6 153294233
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 116u IPv6 153294553
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 117u IPv6 153308598
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 118u IPv6 153299396
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 119u IPv6 153299397
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 120u IPv6 153305498
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 121u IPv6 153305501
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 122u IPv6 153322053
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 123u IPv6 153306847
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 124u IPv6 153306848
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 125u IPv6 153308599
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 126u IPv6 153308072
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 127u IPv6 153308073
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 128u IPv6 153322552
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 129u IPv6 153318287
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 130u IPv6 153318288
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 131u IPv6 153322054
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 132u IPv6 153321874
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 133u IPv6 153321875
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 134u IPv6 153708231
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 135u IPv6 153322490
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 136u IPv6 153322491
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 137u IPv6 153330756
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 138u IPv6 153322687
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 139u IPv6 153322688
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 140u IPv6 153325417
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 141u IPv6 153322873
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 142u IPv6 153322874
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 143u IPv6 153325418
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 144u IPv6 153355133
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 145u IPv6 153325537
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 146u IPv6 153325538
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 147u IPv6 153330757
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 148u IPv6 153330673
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 149u IPv6 153330674
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 150u IPv6 153342009
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 151u IPv6 153342010
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 152u IPv6 153343873
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 153u IPv6 153342462
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 154u IPv6 153342463
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 155u IPv6 153343874
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 156u IPv6 153348641
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 157u IPv6 153344781
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 158u IPv6 153344782
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 159u IPv6 153348642
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 160u IPv6 153354839
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 161u IPv6 153353725
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 162u IPv6 153353726
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 163u IPv6 153354840
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 164u IPv6 153363372
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 165u IPv6 153354979
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 166u IPv6 153354980
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 167u IPv6 153355134
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 168u IPv6 153385927
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 169u IPv6 153445421
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 170u IPv6 153405211
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 171u IPv6 153363371
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 172u IPv6 153405212
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 173u IPv6 153373260
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 174u IPv6 153460225
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 175u IPv6 153449798
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 176u IPv6 153360445
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 177u IPv6 153360459
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 178u IPv6 153449799
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 179u IPv6 153373262
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 180u IPv6 153509124
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 181u IPv6 153411087
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 182u IPv6 153411089
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 183u IPv6 153460228
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 184u IPv6 153428120
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 185u IPv6 153428121
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 187u IPv6 153600041
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 188u IPv6 153557843
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 189u IPv6 153462763
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 190u IPv6 153460417
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 191u IPv6 153460418
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 192u IPv6 153462764
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 193u IPv6 153471908
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 194u IPv6 153468521
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 195u IPv6 153468522
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 196u IPv6 153471980
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 197u IPv6 153557844
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 198u IPv6 153476603
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 199u IPv6 153476604
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 200u IPv6 153508848
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 201u IPv6 153507533
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 202u IPv6 153507534
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 203u IPv6 153508419
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 204u IPv6 153508234
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 205u IPv6 153508235
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 206u IPv6 153508422
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 207u IPv6 153509125
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 209u IPv6 153508845
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 210u IPv6 153508846
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 211u IPv6 153572667
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 212u IPv6 153508849
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 213u IPv6 153516692
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 214u IPv6 153516695
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 216u IPv6 153610112
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 217u IPv6 153610114
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 218u IPv6 153557944
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 219u IPv6 153557945
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 220u IPv6 153572668
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 221u IPv6 153585054
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 222u IPv6 153582445
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 223u IPv6 153582447
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 224u IPv6 153585055
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 225u IPv6 153708561
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 226u IPv6 153594476
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 227u IPv6 153638916
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 229u IPv6 153708562
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 230u IPv6 153600134
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 231u IPv6 153638922
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 232u IPv6 153635793
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 233u IPv6 153708619
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 234u IPv6 153703803
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 235u IPv6 153703804
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 236u IPv6 153604398
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 237u IPv6 153604506
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 238u IPv6 153708578
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 241u IPv6 153635795
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 244r FIFO 0,5
>> 154318314 pipe
>> java 16908 hadoop 245u IPv6 153706355
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 246u IPv6 153706356
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>> java 16908 hadoop 247w FIFO 0,5
>> 154318314 pipe
>> java 16908 hadoop 248u IPv6 153708579
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 249w REG 8,1 15230
>> 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
>> java 16908 hadoop 250u IPv6 153708593
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 251u IPv6 153708594
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 252u IPv6 153708620
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 253u 0000 0,6 0
>> 493 /anon_inode:[eventpoll]
>> java 16908 hadoop 255u IPv6 153742520
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 256u IPv6 153742521
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 259u IPv6 153742533
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 260u IPv6 153742534
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 262u IPv6 153742547
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 263u IPv6 153742548
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 265u IPv6 153742562
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> java 16908 hadoop 266u IPv6 153742563
>> TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010
>> (CLOSE_WAIT)
>> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>>
>>
>>>
>>> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
>>> 12345 is the process id of the hive server?
>>> We should be able to see the names of the files that are open.
>>>
>>> Zheng
>>>
>>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk>
>>> wrote:
>>>
>>>>
>>>> Nope, no luck so far.
>>>>
>>>> We have upped the number of file descriptors and are having to restart
>>>> hive every week or so :(
>>>>
>>>> Any other suggestions would be greatly appreciated.
>>>>
>>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>>
>>>>
>>>>>
>>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>>> connections to 50010 and after a hive restart all is ok again.
>>>>>
>>>>> Andy Kent wrote:
>>>>>
>>>>>>
>>>>>> I can give try and give it a go. I'm not convinced though as we are
>>>>>> working with CSV files and don't touch sequence files at all at the moment.
>>>>>>
>>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and
>>>>>> Hive 0.40
>>>>>>
>>>>>>
>>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile
>>>>>>> where if there were problems opening a file, it would leave a filehandle
>>>>>>> open and never close it.
>>>>>>>
>>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some
>>>>>>> time this week I'll submit it against 0.20.2 -- could you apply this to
>>>>>>> hadoop and let me know if it fixes things for you?
>>>>>>>
>>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth
>>>>>>> <ja...@gmail.com>> wrote:
>>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a
>>>>>>> combination.. maybe if you're doing queries and then closing your thrift
>>>>>>> connection before reading all results, Hive doesn't know what to do and
>>>>>>> leaves the connection open? Once the west coast folks wake up, they might
>>>>>>> have a better answer for you than I do.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent
>>>>>>> <An...@forward.co.uk>> wrote:
>>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting to
>>>>>>>> DFS directly for some reason (maybe for "select *" queries?) and not
>>>>>>>> finishing their reads or closing the connections after.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Thanks for the response.
>>>>>>>
>>>>>>> That's what I was suspecting. I have triple checked and our Ruby code
>>>>>>> and it is defiantly closing it's thrift connections properly.
>>>>>>>
>>>>>>> I'll try running some different queries and see if I can suss out
>>>>>>> some examples of which ones are leaky. Is this something that I should post
>>>>>>> to Jira or is it a known issue? I can't believe other people haven't noticed
>>>>>>> this?
>>>>>>>
>>>>>>>
>>>>>>> <SequenceFile.patch>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>>
>>
>>
>
>
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Bennie Schut <bs...@ebuddy.com>.
I think it's lines like this:
java 16908 hadoop 160u IPv6 153354839 TCP
hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
(ESTABLISHED)
We have the same.
Currently no queries running and many of those connections open. After
running some queries a few more are added to the list.
java 5888 dwh 389u IPv4
1020083623 TCP
victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 390u IPv4
1020083626 TCP
victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 391u IPv4
1020083637 TCP
victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 392u IPv4
1020083641 TCP
victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 393u IPv4
1020083668 TCP
victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 394u IPv4
1020083675 TCP
victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 395u IPv4
1020083678 TCP
victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 396u IPv4
1020083690 TCP
victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 397u IPv4
1020083693 TCP
victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 398u IPv4
1020083703 TCP
victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 399u IPv4
1020083706 TCP
victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 400u IPv4
1020083713 TCP
victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 401u IPv4
1020083716 TCP
victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 402u IPv4
1020083728 TCP
victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 403u IPv4
1020083731 TCP
victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 404u IPv4
1020083740 TCP
victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 405u IPv4
1020083743 TCP
victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 406u IPv4
1020083753 TCP
victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 407u IPv4
1020083756 TCP
victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 408u IPv4
1020083769 TCP
victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 409u IPv4
1020083772 TCP
victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 410u IPv4
1020083781 TCP
victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 411u IPv4
1020083784 TCP
victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 412u IPv4
1020083792 TCP
victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 413u IPv4
1020083795 TCP
victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 414u IPv4
1020083808 TCP
victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 415u IPv4
1020083811 TCP
victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 416u IPv4
1020083820 TCP
victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 417u IPv4
1020083823 TCP
victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 418u IPv4
1020083831 TCP
victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 419u IPv4
1020083834 TCP
victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 420u IPv4
1020083846 TCP
victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 421u IPv4
1020083849 TCP
victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 422u IPv4
1020083865 TCP
victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 423u IPv4
1020083862 TCP
victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 424u IPv4
1020083877 TCP
victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 425u IPv4
1020083880 TCP
victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 426u IPv4
1020083888 TCP
victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 427u IPv4
1020083896 TCP
victoria.ebuddy.com:52095->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 433u IPv4
1020083914 TCP
victoria.ebuddy.com:52102->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 434u IPv4
1020083916 TCP
victoria.ebuddy.com:52103->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 435u IPv4
1020083926 TCP
victoria.ebuddy.com:52107->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 436u IPv4
1020083929 TCP
victoria.ebuddy.com:52108->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 437u IPv4
1020084012 TCP
victoria.ebuddy.com:52148->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 438u IPv4
1020083947 TCP
victoria.ebuddy.com:52115->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 439u IPv4
1020083975 TCP
victoria.ebuddy.com:52132->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 443u IPv4
1020083950 TCP
victoria.ebuddy.com:52116->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 444u IPv4
1020083959 TCP
victoria.ebuddy.com:52123->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 445u IPv4
1020083962 TCP
victoria.ebuddy.com:52127->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 446u IPv4
1020083977 TCP
victoria.ebuddy.com:52133->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 447u IPv4
1020083984 TCP
victoria.ebuddy.com:52136->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 448u IPv4
1020088663 TCP
victoria.ebuddy.com:52176->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 449u IPv4
1020083991 TCP
victoria.ebuddy.com:52139->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 450u IPv4
1020088678 TCP
victoria.ebuddy.com:52181->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 451u IPv4
1020088692 TCP
victoria.ebuddy.com:52188->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 452u IPv4
1020088694 TCP
victoria.ebuddy.com:52189->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 453u IPv4
1020092163 TCP
victoria.ebuddy.com:52198->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 454u IPv4
1020092165 TCP
victoria.ebuddy.com:52199->victoria.ebuddy.com:50010 (ESTABLISHED)
Andy Kent wrote:
> I have included the output from lsof, Unfortunately though I only restarted the server a few mins ago and so it doesn't really illustrate the problem. I'll try to post back with the same command tomorrow so that we can diff between the two.
>
> Thanks, Andy.
>
> java 16908 hadoop cwd DIR 0,17 240 7037865 /var/run/hadoop-0.20
> java 16908 hadoop rtd DIR 8,1 4096 2 /
> java 16908 hadoop txt REG 8,1 50810 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
> java 16908 hadoop mem REG 8,1 127296 3443156 /usr/lib/liblzo2.so.2.0.0
> java 16908 hadoop mem REG 8,1 69472 3735429 /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
> java 16908 hadoop mem REG 8,1 44257 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
> java 16908 hadoop mem REG 8,1 80760 1164352 /lib/libresolv-2.7.so
> java 16908 hadoop mem REG 8,1 22856 1164345 /lib/libnss_dns-2.7.so
> java 16908 hadoop mem REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
> java 16908 hadoop mem REG 8,1 38105 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
> java 16908 hadoop mem REG 8,1 111901 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
> java 16908 hadoop mem REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar
> java 16908 hadoop mem REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
> java 16908 hadoop mem REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
> java 16908 hadoop mem REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
> java 16908 hadoop mem REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar
> java 16908 hadoop mem REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar
> java 16908 hadoop mem REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
> java 16908 hadoop mem REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar
> java 16908 hadoop mem REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
> java 16908 hadoop mem REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
> java 16908 hadoop mem REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar
> java 16908 hadoop mem REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar
> java 16908 hadoop mem REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar
> java 16908 hadoop mem REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar
> java 16908 hadoop mem REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar
> java 16908 hadoop mem REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar
> java 16908 hadoop mem REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar
> java 16908 hadoop mem REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar
> java 16908 hadoop mem REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar
> java 16908 hadoop mem REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar
> java 16908 hadoop mem REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar
> java 16908 hadoop mem REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar
> java 16908 hadoop mem REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
> java 16908 hadoop mem REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
> java 16908 hadoop mem REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
> java 16908 hadoop mem REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
> java 16908 hadoop mem REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
> java 16908 hadoop mem REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
> java 16908 hadoop mem REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
> java 16908 hadoop mem REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
> java 16908 hadoop mem REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
> java 16908 hadoop mem REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar
> java 16908 hadoop mem REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
> java 16908 hadoop mem REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
> java 16908 hadoop mem REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
> java 16908 hadoop mem REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
> java 16908 hadoop mem REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
> java 16908 hadoop mem REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
> java 16908 hadoop mem REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
> java 16908 hadoop mem REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
> java 16908 hadoop mem REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
> java 16908 hadoop mem REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
> java 16908 hadoop mem REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
> java 16908 hadoop mem REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
> java 16908 hadoop mem REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
> java 16908 hadoop mem REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
> java 16908 hadoop mem REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
> java 16908 hadoop mem REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
> java 16908 hadoop mem REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
> java 16908 hadoop mem REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
> java 16908 hadoop mem REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
> java 16908 hadoop mem REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
> java 16908 hadoop mem REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
> java 16908 hadoop mem REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
> java 16908 hadoop mem REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
> java 16908 hadoop mem REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
> java 16908 hadoop mem REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
> java 16908 hadoop mem REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
> java 16908 hadoop mem REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
> java 16908 hadoop mem REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
> java 16908 hadoop mem REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
> java 16908 hadoop mem REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
> java 16908 hadoop mem REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
> java 16908 hadoop mem REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
> java 16908 hadoop mem REG 8,1 25700 3441590 /usr/lib/gconv/gconv-modules.cache
> java 16908 hadoop mem REG 8,1 254076 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
> java 16908 hadoop mem REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
> java 16908 hadoop mem REG 8,1 92117 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
> java 16908 hadoop mem REG 8,1 47528 1164346 /lib/libnss_files-2.7.so
> java 16908 hadoop mem REG 8,1 43480 1164348 /lib/libnss_nis-2.7.so
> java 16908 hadoop mem REG 8,1 35632 1164344 /lib/libnss_compat-2.7.so
> java 16908 hadoop mem REG 8,1 55850 3555581 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
> java 16908 hadoop mem REG 8,1 93080 1164343 /lib/libnsl-2.7.so
> java 16908 hadoop mem REG 8,1 234713 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
> java 16908 hadoop mem REG 8,1 66580 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
> java 16908 hadoop mem REG 8,1 35784 1164353 /lib/librt-2.7.so
> java 16908 hadoop mem REG 8,1 526560 1164341 /lib/libm-2.7.so
> java 16908 hadoop mem REG 8,1 10987257 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
> java 16908 hadoop mem REG 8,1 1436976 1164337 /lib/libc-2.7.so
> java 16908 hadoop mem REG 8,1 14624 1164340 /lib/libdl-2.7.so
> java 16908 hadoop mem REG 8,1 130224 1164351 /lib/libpthread-2.7.so
> java 16908 hadoop mem REG 8,1 127480 1164334 /lib/ld-2.7.so
> java 16908 hadoop mem REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
> java 16908 hadoop mem REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
> java 16908 hadoop mem REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
> java 16908 hadoop mem REG 8,1 32768 3989505 /tmp/hsperfdata_hadoop/16908
> java 16908 hadoop mem REG 8,1 48335 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
> java 16908 hadoop mem REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
> java 16908 hadoop 0u CHR 136,2 4 /dev/pts/2 (deleted)
> java 16908 hadoop 1u CHR 136,2 4 /dev/pts/2 (deleted)
> java 16908 hadoop 2u CHR 136,2 4 /dev/pts/2 (deleted)
> java 16908 hadoop 3r REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
> java 16908 hadoop 4r REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
> java 16908 hadoop 5r REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
> java 16908 hadoop 6r REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
> java 16908 hadoop 7r REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
> java 16908 hadoop 8r REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
> java 16908 hadoop 9r REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
> java 16908 hadoop 10r REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
> java 16908 hadoop 11r REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
> java 16908 hadoop 12r REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
> java 16908 hadoop 13r REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
> java 16908 hadoop 14r REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
> java 16908 hadoop 15r REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
> java 16908 hadoop 16r REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
> java 16908 hadoop 17r REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
> java 16908 hadoop 18r REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
> java 16908 hadoop 19r REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
> java 16908 hadoop 20r REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
> java 16908 hadoop 21r REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
> java 16908 hadoop 22r REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
> java 16908 hadoop 23r REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
> java 16908 hadoop 24r REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
> java 16908 hadoop 25r REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
> java 16908 hadoop 26r REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
> java 16908 hadoop 27r REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
> java 16908 hadoop 28r REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
> java 16908 hadoop 29r REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
> java 16908 hadoop 30r REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
> java 16908 hadoop 31r REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
> java 16908 hadoop 32r REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
> java 16908 hadoop 33r REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
> java 16908 hadoop 34r REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
> java 16908 hadoop 35r REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
> java 16908 hadoop 36r REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
> java 16908 hadoop 37r REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
> java 16908 hadoop 38r REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
> java 16908 hadoop 39r REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar
> java 16908 hadoop 40r REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
> java 16908 hadoop 41r REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
> java 16908 hadoop 42r REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
> java 16908 hadoop 43r REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
> java 16908 hadoop 44r REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
> java 16908 hadoop 45r REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
> java 16908 hadoop 46r REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
> java 16908 hadoop 47r REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
> java 16908 hadoop 48r REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
> java 16908 hadoop 49r REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar
> java 16908 hadoop 50r REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar
> java 16908 hadoop 51r REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar
> java 16908 hadoop 52r REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar
> java 16908 hadoop 53r REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar
> java 16908 hadoop 54r REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar
> java 16908 hadoop 55r REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar
> java 16908 hadoop 56r REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar
> java 16908 hadoop 57r REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar
> java 16908 hadoop 58r REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar
> java 16908 hadoop 59r REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar
> java 16908 hadoop 60r REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar
> java 16908 hadoop 61r REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
> java 16908 hadoop 62r REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
> java 16908 hadoop 63r REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar
> java 16908 hadoop 64r REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
> java 16908 hadoop 65r REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar
> java 16908 hadoop 66r REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar
> java 16908 hadoop 67r REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
> java 16908 hadoop 68r REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
> java 16908 hadoop 69r REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
> java 16908 hadoop 70r REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar
> java 16908 hadoop 71r CHR 1,8 1818 /dev/random
> java 16908 hadoop 72r CHR 1,9 1824 /dev/urandom
> java 16908 hadoop 73u IPv6 153142741 TCP *:webmin (LISTEN)
> java 16908 hadoop 74u sock 0,4 153142739 can't identify protocol
> java 16908 hadoop 76w REG 8,1 6867 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
> java 16908 hadoop 77u IPv6 154300915 TCP hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 78r REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
> java 16908 hadoop 79u IPv6 153477399 TCP hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 80u IPv6 153594480 TCP hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 81w REG 8,1 15491 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
> java 16908 hadoop 82u IPv6 153177590 TCP hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 83r REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
> java 16908 hadoop 84u IPv6 153177591 TCP hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 85u unix 0xffff8101b5090000 153143422 socket
> java 16908 hadoop 86w REG 8,1 62510 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
> java 16908 hadoop 87w REG 8,1 960 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
> java 16908 hadoop 88u IPv6 153219191 TCP hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 89u IPv6 153143432 TCP hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 90u IPv6 153143434 TCP hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 91u IPv6 153219192 TCP hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 92u IPv6 153477400 TCP hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 93u IPv6 153445420 TCP hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 94u IPv6 153385926 TCP hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 95u IPv6 153322551 TCP hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 96u IPv6 153220498 TCP hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 97u IPv6 153220499 TCP hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 98u IPv6 153273816 TCP hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 99u IPv6 153273819 TCP hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 100u IPv6 153293338 TCP hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 102u IPv6 153708229 TCP hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 103u IPv6 153276086 TCP hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 104u IPv6 153276087 TCP hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 105u IPv6 153291015 TCP hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 106u IPv6 153291016 TCP hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 107u IPv6 153293339 TCP hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 108u IPv6 153291444 TCP hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 109u IPv6 153291445 TCP hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 110u IPv6 153294552 TCP hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 111u IPv6 153294209 TCP hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 112u IPv6 153294210 TCP hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 113w REG 8,1 11562 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
> java 16908 hadoop 114u IPv6 153294232 TCP hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 115u IPv6 153294233 TCP hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 116u IPv6 153294553 TCP hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 117u IPv6 153308598 TCP hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 118u IPv6 153299396 TCP hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 119u IPv6 153299397 TCP hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 120u IPv6 153305498 TCP hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 121u IPv6 153305501 TCP hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 122u IPv6 153322053 TCP hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 123u IPv6 153306847 TCP hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 124u IPv6 153306848 TCP hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 125u IPv6 153308599 TCP hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 126u IPv6 153308072 TCP hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 127u IPv6 153308073 TCP hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 128u IPv6 153322552 TCP hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 129u IPv6 153318287 TCP hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 130u IPv6 153318288 TCP hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 131u IPv6 153322054 TCP hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 132u IPv6 153321874 TCP hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 133u IPv6 153321875 TCP hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 134u IPv6 153708231 TCP hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 135u IPv6 153322490 TCP hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 136u IPv6 153322491 TCP hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 137u IPv6 153330756 TCP hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 138u IPv6 153322687 TCP hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 139u IPv6 153322688 TCP hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 140u IPv6 153325417 TCP hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 141u IPv6 153322873 TCP hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 142u IPv6 153322874 TCP hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 143u IPv6 153325418 TCP hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 144u IPv6 153355133 TCP hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 145u IPv6 153325537 TCP hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 146u IPv6 153325538 TCP hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 147u IPv6 153330757 TCP hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 148u IPv6 153330673 TCP hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 149u IPv6 153330674 TCP hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 150u IPv6 153342009 TCP hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 151u IPv6 153342010 TCP hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 152u IPv6 153343873 TCP hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 153u IPv6 153342462 TCP hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 154u IPv6 153342463 TCP hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 155u IPv6 153343874 TCP hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 156u IPv6 153348641 TCP hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 157u IPv6 153344781 TCP hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 158u IPv6 153344782 TCP hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 159u IPv6 153348642 TCP hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 160u IPv6 153354839 TCP hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 161u IPv6 153353725 TCP hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 162u IPv6 153353726 TCP hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 163u IPv6 153354840 TCP hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 164u IPv6 153363372 TCP hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 165u IPv6 153354979 TCP hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 166u IPv6 153354980 TCP hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 167u IPv6 153355134 TCP hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 168u IPv6 153385927 TCP hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 169u IPv6 153445421 TCP hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 170u IPv6 153405211 TCP hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 171u IPv6 153363371 TCP hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 172u IPv6 153405212 TCP hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 173u IPv6 153373260 TCP hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 174u IPv6 153460225 TCP hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 175u IPv6 153449798 TCP hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 176u IPv6 153360445 TCP hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 177u IPv6 153360459 TCP hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 178u IPv6 153449799 TCP hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 179u IPv6 153373262 TCP hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 180u IPv6 153509124 TCP hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 181u IPv6 153411087 TCP hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 182u IPv6 153411089 TCP hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 183u IPv6 153460228 TCP hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 184u IPv6 153428120 TCP hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 185u IPv6 153428121 TCP hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 187u IPv6 153600041 TCP hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 188u IPv6 153557843 TCP hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 189u IPv6 153462763 TCP hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 190u IPv6 153460417 TCP hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 191u IPv6 153460418 TCP hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 192u IPv6 153462764 TCP hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 193u IPv6 153471908 TCP hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 194u IPv6 153468521 TCP hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 195u IPv6 153468522 TCP hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 196u IPv6 153471980 TCP hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 197u IPv6 153557844 TCP hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 198u IPv6 153476603 TCP hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 199u IPv6 153476604 TCP hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 200u IPv6 153508848 TCP hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 201u IPv6 153507533 TCP hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 202u IPv6 153507534 TCP hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 203u IPv6 153508419 TCP hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 204u IPv6 153508234 TCP hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 205u IPv6 153508235 TCP hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 206u IPv6 153508422 TCP hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 207u IPv6 153509125 TCP hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 209u IPv6 153508845 TCP hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 210u IPv6 153508846 TCP hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 211u IPv6 153572667 TCP hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 212u IPv6 153508849 TCP hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 213u IPv6 153516692 TCP hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 214u IPv6 153516695 TCP hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 216u IPv6 153610112 TCP hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 217u IPv6 153610114 TCP hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 218u IPv6 153557944 TCP hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 219u IPv6 153557945 TCP hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 220u IPv6 153572668 TCP hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 221u IPv6 153585054 TCP hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 222u IPv6 153582445 TCP hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 223u IPv6 153582447 TCP hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 224u IPv6 153585055 TCP hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 225u IPv6 153708561 TCP hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 226u IPv6 153594476 TCP hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 227u IPv6 153638916 TCP hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 229u IPv6 153708562 TCP hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 230u IPv6 153600134 TCP hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 231u IPv6 153638922 TCP hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 232u IPv6 153635793 TCP hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 233u IPv6 153708619 TCP hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 234u IPv6 153703803 TCP hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 235u IPv6 153703804 TCP hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 236u IPv6 153604398 TCP hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 237u IPv6 153604506 TCP hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 238u IPv6 153708578 TCP hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 241u IPv6 153635795 TCP hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 244r FIFO 0,5 154318314 pipe
> java 16908 hadoop 245u IPv6 153706355 TCP hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 246u IPv6 153706356 TCP hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
> java 16908 hadoop 247w FIFO 0,5 154318314 pipe
> java 16908 hadoop 248u IPv6 153708579 TCP hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 249w REG 8,1 15230 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
> java 16908 hadoop 250u IPv6 153708593 TCP hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 251u IPv6 153708594 TCP hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 252u IPv6 153708620 TCP hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 253u 0000 0,6 0 493 /anon_inode:[eventpoll]
> java 16908 hadoop 255u IPv6 153742520 TCP hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 256u IPv6 153742521 TCP hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 259u IPv6 153742533 TCP hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 260u IPv6 153742534 TCP hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 262u IPv6 153742547 TCP hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 263u IPv6 153742548 TCP hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 265u IPv6 153742562 TCP hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> java 16908 hadoop 266u IPv6 153742563 TCP hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
> On 15 Feb 2010, at 22:42, Zheng Shao wrote:
>
>
>> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
>> 12345 is the process id of the hive server?
>> We should be able to see the names of the files that are open.
>>
>> Zheng
>>
>> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk> wrote:
>>
>>> Nope, no luck so far.
>>>
>>> We have upped the number of file descriptors and are having to restart hive every week or so :(
>>>
>>> Any other suggestions would be greatly appreciated.
>>>
>>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>>
>>>
>>>> Did this help? I'm running into a similar problem. slowly leaking
>>>> connections to 50010 and after a hive restart all is ok again.
>>>>
>>>> Andy Kent wrote:
>>>>
>>>>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>>>>
>>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>>>>
>>>>>
>>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>>>>
>>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>>>>
>>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <ja...@gmail.com>> wrote:
>>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <An...@forward.co.uk>> wrote:
>>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>>>>
>>>>>>>
>>>>>> Thanks for the response.
>>>>>>
>>>>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>>>>
>>>>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>>>>
>>>>>>
>>>>>> <SequenceFile.patch>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Yours,
>> Zheng
>>
>>
>
>
Re: Hive Server Leaking File Descriptors?
Posted by Andy Kent <An...@forward.co.uk>.
I have included the output from lsof, Unfortunately though I only restarted the server a few mins ago and so it doesn't really illustrate the problem. I'll try to post back with the same command tomorrow so that we can diff between the two.
Thanks, Andy.
java 16908 hadoop cwd DIR 0,17 240 7037865 /var/run/hadoop-0.20
java 16908 hadoop rtd DIR 8,1 4096 2 /
java 16908 hadoop txt REG 8,1 50810 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
java 16908 hadoop mem REG 8,1 127296 3443156 /usr/lib/liblzo2.so.2.0.0
java 16908 hadoop mem REG 8,1 69472 3735429 /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
java 16908 hadoop mem REG 8,1 44257 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
java 16908 hadoop mem REG 8,1 80760 1164352 /lib/libresolv-2.7.so
java 16908 hadoop mem REG 8,1 22856 1164345 /lib/libnss_dns-2.7.so
java 16908 hadoop mem REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
java 16908 hadoop mem REG 8,1 38105 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
java 16908 hadoop mem REG 8,1 111901 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
java 16908 hadoop mem REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar
java 16908 hadoop mem REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
java 16908 hadoop mem REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
java 16908 hadoop mem REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
java 16908 hadoop mem REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar
java 16908 hadoop mem REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar
java 16908 hadoop mem REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
java 16908 hadoop mem REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar
java 16908 hadoop mem REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
java 16908 hadoop mem REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
java 16908 hadoop mem REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar
java 16908 hadoop mem REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar
java 16908 hadoop mem REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar
java 16908 hadoop mem REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar
java 16908 hadoop mem REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar
java 16908 hadoop mem REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar
java 16908 hadoop mem REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar
java 16908 hadoop mem REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar
java 16908 hadoop mem REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar
java 16908 hadoop mem REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar
java 16908 hadoop mem REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar
java 16908 hadoop mem REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar
java 16908 hadoop mem REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
java 16908 hadoop mem REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
java 16908 hadoop mem REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
java 16908 hadoop mem REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
java 16908 hadoop mem REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
java 16908 hadoop mem REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
java 16908 hadoop mem REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
java 16908 hadoop mem REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
java 16908 hadoop mem REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
java 16908 hadoop mem REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar
java 16908 hadoop mem REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
java 16908 hadoop mem REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
java 16908 hadoop mem REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
java 16908 hadoop mem REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
java 16908 hadoop mem REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
java 16908 hadoop mem REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
java 16908 hadoop mem REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
java 16908 hadoop mem REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
java 16908 hadoop mem REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
java 16908 hadoop mem REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
java 16908 hadoop mem REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
java 16908 hadoop mem REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
java 16908 hadoop mem REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
java 16908 hadoop mem REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
java 16908 hadoop mem REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
java 16908 hadoop mem REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
java 16908 hadoop mem REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
java 16908 hadoop mem REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
java 16908 hadoop mem REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
java 16908 hadoop mem REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
java 16908 hadoop mem REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
java 16908 hadoop mem REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
java 16908 hadoop mem REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
java 16908 hadoop mem REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
java 16908 hadoop mem REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
java 16908 hadoop mem REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
java 16908 hadoop mem REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
java 16908 hadoop mem REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
java 16908 hadoop mem REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
java 16908 hadoop mem REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
java 16908 hadoop mem REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
java 16908 hadoop mem REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
java 16908 hadoop mem REG 8,1 25700 3441590 /usr/lib/gconv/gconv-modules.cache
java 16908 hadoop mem REG 8,1 254076 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE
java 16908 hadoop mem REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
java 16908 hadoop mem REG 8,1 92117 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so
java 16908 hadoop mem REG 8,1 47528 1164346 /lib/libnss_files-2.7.so
java 16908 hadoop mem REG 8,1 43480 1164348 /lib/libnss_nis-2.7.so
java 16908 hadoop mem REG 8,1 35632 1164344 /lib/libnss_compat-2.7.so
java 16908 hadoop mem REG 8,1 55850 3555581 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so
java 16908 hadoop mem REG 8,1 93080 1164343 /lib/libnsl-2.7.so
java 16908 hadoop mem REG 8,1 234713 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so
java 16908 hadoop mem REG 8,1 66580 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so
java 16908 hadoop mem REG 8,1 35784 1164353 /lib/librt-2.7.so
java 16908 hadoop mem REG 8,1 526560 1164341 /lib/libm-2.7.so
java 16908 hadoop mem REG 8,1 10987257 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so
java 16908 hadoop mem REG 8,1 1436976 1164337 /lib/libc-2.7.so
java 16908 hadoop mem REG 8,1 14624 1164340 /lib/libdl-2.7.so
java 16908 hadoop mem REG 8,1 130224 1164351 /lib/libpthread-2.7.so
java 16908 hadoop mem REG 8,1 127480 1164334 /lib/ld-2.7.so
java 16908 hadoop mem REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
java 16908 hadoop mem REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
java 16908 hadoop mem REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
java 16908 hadoop mem REG 8,1 32768 3989505 /tmp/hsperfdata_hadoop/16908
java 16908 hadoop mem REG 8,1 48335 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so
java 16908 hadoop mem REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
java 16908 hadoop 0u CHR 136,2 4 /dev/pts/2 (deleted)
java 16908 hadoop 1u CHR 136,2 4 /dev/pts/2 (deleted)
java 16908 hadoop 2u CHR 136,2 4 /dev/pts/2 (deleted)
java 16908 hadoop 3r REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar
java 16908 hadoop 4r REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar
java 16908 hadoop 5r REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar
java 16908 hadoop 6r REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar
java 16908 hadoop 7r REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar
java 16908 hadoop 8r REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar
java 16908 hadoop 9r REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar
java 16908 hadoop 10r REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar
java 16908 hadoop 11r REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar
java 16908 hadoop 12r REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar
java 16908 hadoop 13r REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar
java 16908 hadoop 14r REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar
java 16908 hadoop 15r REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar
java 16908 hadoop 16r REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar
java 16908 hadoop 17r REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar
java 16908 hadoop 18r REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar
java 16908 hadoop 19r REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar
java 16908 hadoop 20r REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar
java 16908 hadoop 21r REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar
java 16908 hadoop 22r REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar
java 16908 hadoop 23r REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar
java 16908 hadoop 24r REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar
java 16908 hadoop 25r REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar
java 16908 hadoop 26r REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar
java 16908 hadoop 27r REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar
java 16908 hadoop 28r REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar
java 16908 hadoop 29r REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar
java 16908 hadoop 30r REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar
java 16908 hadoop 31r REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar
java 16908 hadoop 32r REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar
java 16908 hadoop 33r REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar
java 16908 hadoop 34r REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
java 16908 hadoop 35r REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
java 16908 hadoop 36r REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
java 16908 hadoop 37r REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
java 16908 hadoop 38r REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
java 16908 hadoop 39r REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar
java 16908 hadoop 40r REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
java 16908 hadoop 41r REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
java 16908 hadoop 42r REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
java 16908 hadoop 43r REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
java 16908 hadoop 44r REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
java 16908 hadoop 45r REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
java 16908 hadoop 46r REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
java 16908 hadoop 47r REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
java 16908 hadoop 48r REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
java 16908 hadoop 49r REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar
java 16908 hadoop 50r REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar
java 16908 hadoop 51r REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar
java 16908 hadoop 52r REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar
java 16908 hadoop 53r REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar
java 16908 hadoop 54r REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar
java 16908 hadoop 55r REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar
java 16908 hadoop 56r REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar
java 16908 hadoop 57r REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar
java 16908 hadoop 58r REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar
java 16908 hadoop 59r REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar
java 16908 hadoop 60r REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar
java 16908 hadoop 61r REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
java 16908 hadoop 62r REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
java 16908 hadoop 63r REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar
java 16908 hadoop 64r REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
java 16908 hadoop 65r REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar
java 16908 hadoop 66r REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar
java 16908 hadoop 67r REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
java 16908 hadoop 68r REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
java 16908 hadoop 69r REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
java 16908 hadoop 70r REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar
java 16908 hadoop 71r CHR 1,8 1818 /dev/random
java 16908 hadoop 72r CHR 1,9 1824 /dev/urandom
java 16908 hadoop 73u IPv6 153142741 TCP *:webmin (LISTEN)
java 16908 hadoop 74u sock 0,4 153142739 can't identify protocol
java 16908 hadoop 76w REG 8,1 6867 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt
java 16908 hadoop 77u IPv6 154300915 TCP hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 78r REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
java 16908 hadoop 79u IPv6 153477399 TCP hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 80u IPv6 153594480 TCP hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 81w REG 8,1 15491 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt
java 16908 hadoop 82u IPv6 153177590 TCP hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 83r REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar
java 16908 hadoop 84u IPv6 153177591 TCP hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 85u unix 0xffff8101b5090000 153143422 socket
java 16908 hadoop 86w REG 8,1 62510 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt
java 16908 hadoop 87w REG 8,1 960 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt
java 16908 hadoop 88u IPv6 153219191 TCP hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 89u IPv6 153143432 TCP hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 90u IPv6 153143434 TCP hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 91u IPv6 153219192 TCP hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 92u IPv6 153477400 TCP hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 93u IPv6 153445420 TCP hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 94u IPv6 153385926 TCP hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 95u IPv6 153322551 TCP hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 96u IPv6 153220498 TCP hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 97u IPv6 153220499 TCP hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 98u IPv6 153273816 TCP hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 99u IPv6 153273819 TCP hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 100u IPv6 153293338 TCP hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 102u IPv6 153708229 TCP hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 103u IPv6 153276086 TCP hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 104u IPv6 153276087 TCP hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 105u IPv6 153291015 TCP hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 106u IPv6 153291016 TCP hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 107u IPv6 153293339 TCP hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 108u IPv6 153291444 TCP hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 109u IPv6 153291445 TCP hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 110u IPv6 153294552 TCP hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 111u IPv6 153294209 TCP hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 112u IPv6 153294210 TCP hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 113w REG 8,1 11562 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt
java 16908 hadoop 114u IPv6 153294232 TCP hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 115u IPv6 153294233 TCP hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 116u IPv6 153294553 TCP hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 117u IPv6 153308598 TCP hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 118u IPv6 153299396 TCP hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 119u IPv6 153299397 TCP hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 120u IPv6 153305498 TCP hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 121u IPv6 153305501 TCP hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 122u IPv6 153322053 TCP hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 123u IPv6 153306847 TCP hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 124u IPv6 153306848 TCP hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 125u IPv6 153308599 TCP hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 126u IPv6 153308072 TCP hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 127u IPv6 153308073 TCP hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 128u IPv6 153322552 TCP hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 129u IPv6 153318287 TCP hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 130u IPv6 153318288 TCP hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 131u IPv6 153322054 TCP hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 132u IPv6 153321874 TCP hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 133u IPv6 153321875 TCP hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 134u IPv6 153708231 TCP hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 135u IPv6 153322490 TCP hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 136u IPv6 153322491 TCP hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 137u IPv6 153330756 TCP hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 138u IPv6 153322687 TCP hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 139u IPv6 153322688 TCP hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 140u IPv6 153325417 TCP hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 141u IPv6 153322873 TCP hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 142u IPv6 153322874 TCP hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 143u IPv6 153325418 TCP hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 144u IPv6 153355133 TCP hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 145u IPv6 153325537 TCP hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 146u IPv6 153325538 TCP hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 147u IPv6 153330757 TCP hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 148u IPv6 153330673 TCP hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 149u IPv6 153330674 TCP hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 150u IPv6 153342009 TCP hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 151u IPv6 153342010 TCP hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 152u IPv6 153343873 TCP hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 153u IPv6 153342462 TCP hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 154u IPv6 153342463 TCP hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 155u IPv6 153343874 TCP hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 156u IPv6 153348641 TCP hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 157u IPv6 153344781 TCP hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 158u IPv6 153344782 TCP hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 159u IPv6 153348642 TCP hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 160u IPv6 153354839 TCP hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 161u IPv6 153353725 TCP hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 162u IPv6 153353726 TCP hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 163u IPv6 153354840 TCP hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 164u IPv6 153363372 TCP hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 165u IPv6 153354979 TCP hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 166u IPv6 153354980 TCP hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 167u IPv6 153355134 TCP hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 168u IPv6 153385927 TCP hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 169u IPv6 153445421 TCP hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 170u IPv6 153405211 TCP hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 171u IPv6 153363371 TCP hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 172u IPv6 153405212 TCP hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 173u IPv6 153373260 TCP hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 174u IPv6 153460225 TCP hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 175u IPv6 153449798 TCP hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 176u IPv6 153360445 TCP hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 177u IPv6 153360459 TCP hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 178u IPv6 153449799 TCP hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 179u IPv6 153373262 TCP hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 180u IPv6 153509124 TCP hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 181u IPv6 153411087 TCP hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 182u IPv6 153411089 TCP hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 183u IPv6 153460228 TCP hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 184u IPv6 153428120 TCP hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 185u IPv6 153428121 TCP hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 187u IPv6 153600041 TCP hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 188u IPv6 153557843 TCP hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 189u IPv6 153462763 TCP hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 190u IPv6 153460417 TCP hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 191u IPv6 153460418 TCP hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 192u IPv6 153462764 TCP hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 193u IPv6 153471908 TCP hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 194u IPv6 153468521 TCP hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 195u IPv6 153468522 TCP hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 196u IPv6 153471980 TCP hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 197u IPv6 153557844 TCP hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 198u IPv6 153476603 TCP hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 199u IPv6 153476604 TCP hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 200u IPv6 153508848 TCP hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 201u IPv6 153507533 TCP hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 202u IPv6 153507534 TCP hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 203u IPv6 153508419 TCP hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 204u IPv6 153508234 TCP hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 205u IPv6 153508235 TCP hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 206u IPv6 153508422 TCP hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 207u IPv6 153509125 TCP hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 209u IPv6 153508845 TCP hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 210u IPv6 153508846 TCP hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 211u IPv6 153572667 TCP hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 212u IPv6 153508849 TCP hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 213u IPv6 153516692 TCP hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 214u IPv6 153516695 TCP hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 216u IPv6 153610112 TCP hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 217u IPv6 153610114 TCP hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 218u IPv6 153557944 TCP hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 219u IPv6 153557945 TCP hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 220u IPv6 153572668 TCP hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 221u IPv6 153585054 TCP hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 222u IPv6 153582445 TCP hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 223u IPv6 153582447 TCP hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 224u IPv6 153585055 TCP hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 225u IPv6 153708561 TCP hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 226u IPv6 153594476 TCP hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 227u IPv6 153638916 TCP hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 229u IPv6 153708562 TCP hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 230u IPv6 153600134 TCP hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 231u IPv6 153638922 TCP hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 232u IPv6 153635793 TCP hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 233u IPv6 153708619 TCP hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 234u IPv6 153703803 TCP hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 235u IPv6 153703804 TCP hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 236u IPv6 153604398 TCP hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 237u IPv6 153604506 TCP hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 238u IPv6 153708578 TCP hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 241u IPv6 153635795 TCP hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 244r FIFO 0,5 154318314 pipe
java 16908 hadoop 245u IPv6 153706355 TCP hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 246u IPv6 153706356 TCP hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED)
java 16908 hadoop 247w FIFO 0,5 154318314 pipe
java 16908 hadoop 248u IPv6 153708579 TCP hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 249w REG 8,1 15230 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt
java 16908 hadoop 250u IPv6 153708593 TCP hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 251u IPv6 153708594 TCP hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 252u IPv6 153708620 TCP hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 253u 0000 0,6 0 493 /anon_inode:[eventpoll]
java 16908 hadoop 255u IPv6 153742520 TCP hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 256u IPv6 153742521 TCP hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 259u IPv6 153742533 TCP hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 260u IPv6 153742534 TCP hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 262u IPv6 153742547 TCP hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 263u IPv6 153742548 TCP hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 265u IPv6 153742562 TCP hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
java 16908 hadoop 266u IPv6 153742563 TCP hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT)
On 15 Feb 2010, at 22:42, Zheng Shao wrote:
> Can you go to that box, sudo as root, and do "lsof | grep 12345" where
> 12345 is the process id of the hive server?
> We should be able to see the names of the files that are open.
>
> Zheng
>
> On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk> wrote:
>> Nope, no luck so far.
>>
>> We have upped the number of file descriptors and are having to restart hive every week or so :(
>>
>> Any other suggestions would be greatly appreciated.
>>
>> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>>
>>> Did this help? I'm running into a similar problem. slowly leaking
>>> connections to 50010 and after a hive restart all is ok again.
>>>
>>> Andy Kent wrote:
>>>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>>>
>>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>>>
>>>>
>>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>>
>>>>
>>>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>>>
>>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>>>
>>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <ja...@gmail.com>> wrote:
>>>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>>>
>>>>>
>>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <An...@forward.co.uk>> wrote:
>>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>>
>>>>>
>>>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>>>
>>>>> Thanks for the response.
>>>>>
>>>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>>>
>>>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>>>
>>>>>
>>>>> <SequenceFile.patch>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
> --
> Yours,
> Zheng
>
Re: Hive Server Leaking File Descriptors?
Posted by Zheng Shao <zs...@gmail.com>.
Can you go to that box, sudo as root, and do "lsof | grep 12345" where
12345 is the process id of the hive server?
We should be able to see the names of the files that are open.
Zheng
On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <An...@forward.co.uk> wrote:
> Nope, no luck so far.
>
> We have upped the number of file descriptors and are having to restart hive every week or so :(
>
> Any other suggestions would be greatly appreciated.
>
> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>
>> Did this help? I'm running into a similar problem. slowly leaking
>> connections to 50010 and after a hive restart all is ok again.
>>
>> Andy Kent wrote:
>>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>>
>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>>
>>>
>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>
>>>
>>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>>
>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>>
>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <ja...@gmail.com>> wrote:
>>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>>
>>>>
>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <An...@forward.co.uk>> wrote:
>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>
>>>>
>>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>>
>>>> Thanks for the response.
>>>>
>>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>>
>>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>>
>>>>
>>>> <SequenceFile.patch>
>>>>
>>>
>>>
>>
>>
>
>
--
Yours,
Zheng
Re: Hive Server Leaking File Descriptors?
Posted by Andy Kent <An...@forward.co.uk>.
Nope, no luck so far.
We have upped the number of file descriptors and are having to restart hive every week or so :(
Any other suggestions would be greatly appreciated.
On 15 Feb 2010, at 14:09, Bennie Schut wrote:
> Did this help? I'm running into a similar problem. slowly leaking
> connections to 50010 and after a hive restart all is ok again.
>
> Andy Kent wrote:
>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>
>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>
>>
>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>
>>
>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>
>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>
>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <ja...@gmail.com>> wrote:
>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>
>>>
>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <An...@forward.co.uk>> wrote:
>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>
>>>
>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>
>>> Thanks for the response.
>>>
>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>
>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>
>>>
>>> <SequenceFile.patch>
>>>
>>
>>
>
>