You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Roman Shaposhnik <ro...@shaposhnik.org> on 2014/02/12 00:10:23 UTC
GIRAPH-825 and GIRAPH-840
Hi!
Given how big the diff here are:
https://issues.apache.org/jira/browse/GIRAPH-825
https://issues.apache.org/jira/browse/GIRAPH-840
I am wondering whether it is realistic
to have them in 1.1.0.
Would appreciate folks chiming in.
Thanks,
Roman.
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
Hi Sebastian,
On Thu, Feb 20, 2014 at 11:09:54AM +0100, Sebastian Schelter wrote:
> Hi Armando & Claudio,
>
> I tested your latest patch, unfortunately it still doesn't work for me.
I was also able to run the computation on a cluster and noticed that
something is going on. Thank you for the stacktrace, that will be
helpful. I'll let you know as soon as I am able to come up with a newer
patch that fixes something.
Cheers,
Armando
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Claudio,
I reran the job with giraph.partitionsDirectory=/tmp/ which should be
writable for everyone. I also checked the diskspace for the directory on
a few machines, they all had > 100GB of free space.
I'm still getting the same error though unfortunately.
--sebastian
On 02/25/2014 09:02 AM, Claudio Martella wrote:
> I have the feeling that an IOException is being swallowed somewhere. For
> example, Sebastian, are you sure that you have enough space locally and
> permissions? I'll investigate this.
>
>
> On Thu, Feb 20, 2014 at 11:09 AM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> Hi Armando & Claudio,
>>
>> I tested your latest patch, unfortunately it still doesn't work for me.
>>
>> I applied the patch to the current trunk, copied my algorithm code and ran
>> it on the cluster. If it runs in memory, everything works well. If I enable
>> out-of-core I run into the deadlock as before.
>>
>> Here's the command I used:
>>
>> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.hyperball.HyperBall
>> --vertexInputFormat org.apache.giraph.examples.hyperball.HyperBallTextInputFormat
>> --vertexInputPath hdfs:///ssc/grades/data/twitter-negative/
>> --vertexOutputFormat org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>> --outputPath hdfs:///ssc/tmp-123/ --combiner org.apache.giraph.examples.hyperball.HyperLogLogCombiner
>> --outEdges org.apache.giraph.edge.LongNullArrayEdges --workers 24
>> --customArguments giraph.oneToAllMsgSending=true,giraph.isStaticGraph=
>> true,giraph.numComputeThreads=15,giraph.numInputThreads=15,
>> giraph.numOutputThreads=15,giraph.maxNumberOfSupersteps=
>> 30,giraph.useOutOfCoreGraph=true,giraph.maxPartitionsInMemory=20
>>
>> Best,
>> Sebastian
>>
>> Here's a stracktrace of one workers:
>>
>> 2014-02-20 11:05:17
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):
>>
>> "Attach Listener" daemon prio=10 tid=0x00007f98f0096000 nid=0x78cb waiting
>> on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "netty-server-worker-15" prio=10 tid=0x00007f9910025000 nid=0x77b3
>> runnable [0x00007f9914aa1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000adfc3140> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000adfc3218> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000adfc3040> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-14" prio=10 tid=0x00007f9910023000 nid=0x77b2
>> runnable [0x00007f9914ba2000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae3a0680> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae22fae8> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae3a0590> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-13" prio=10 tid=0x00007f9910021000 nid=0x77b1
>> runnable [0x00007f9914ca3000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae474e30> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae4780c8> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae474d40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-12" prio=10 tid=0x00007f991001f800 nid=0x77b0
>> runnable [0x00007f9914da4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000adfee950> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000adfeec20> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000adfee860> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-11" prio=10 tid=0x00007f991001d800 nid=0x77af
>> runnable [0x00007f9914ea5000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae0fc9b0> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae0fcc80> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae0fc8c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-10" prio=10 tid=0x00007f991001c000 nid=0x77ae waiting
>> for monitor entry [0x00007f9914fa6000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-9" prio=10 tid=0x00007f991001a800 nid=0x77ad runnable
>> [0x00007f99150a7000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000adf80e78> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000adf81148> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000adf80d88> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-7" prio=10 tid=0x00007f98f8015800 nid=0x77ac waiting on
>> condition [0x00007f99151a8000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae444d38> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-8" prio=10 tid=0x00007f9910018800 nid=0x77ab runnable
>> [0x00007f99152a9000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae0fd200> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae0fd2d8> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae0fd100> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-7" prio=10 tid=0x00007f9910016800 nid=0x77aa waiting
>> for monitor entry [0x00007f99153aa000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-6" prio=10 tid=0x00007f9928452800 nid=0x77a9 waiting on
>> condition [0x00007f99154ab000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae4466a0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-5" prio=10 tid=0x00007f99006d6000 nid=0x77a8 waiting on
>> condition [0x00007f99155ac000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae44df48> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-6" prio=10 tid=0x00007f9910014800 nid=0x77a7 waiting
>> for monitor entry [0x00007f99156ad000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-4" prio=10 tid=0x00007f9908661800 nid=0x77a6 waiting on
>> condition [0x00007f99157ae000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae4287a0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-3" prio=10 tid=0x00007f98f0005800 nid=0x77a5 waiting on
>> condition [0x00007f99158af000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae452518> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-5" prio=10 tid=0x00007f9910012800 nid=0x77a4 waiting
>> for monitor entry [0x00007f99159b0000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-4" prio=10 tid=0x00007f9910010800 nid=0x77a3 runnable
>> [0x00007f9915ab1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000adfadba0> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000adfb86d8> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000adfadab0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-3" prio=10 tid=0x00007f991000e800 nid=0x77a2 runnable
>> [0x00007f9915bb1000]
>> java.lang.Thread.State: RUNNABLE
>> at java.io.FileInputStream.readBytes(Native Method)
>> at java.io.FileInputStream.read(FileInputStream.java:242)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked <0x00000005be8e5778> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>> at org.apache.giraph.edge.LongNullArrayEdges.readFields(
>> LongNullArrayEdges.java:161)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> readOutEdges(DiskBackedPartitionStore.java:387)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> loadPartition(DiskBackedPartitionStore.java:441)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getPartition(DiskBackedPartitionStore.java:771)
>> - locked <0x00000000ae3c8458> (a org.apache.giraph.partition.
>> DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:238)
>> - locked <0x00000000ae3c8458> (a org.apache.giraph.partition.
>> DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-2" prio=10 tid=0x0000000000eb2800 nid=0x77a1 waiting on
>> condition [0x00007f9915cb3000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae453e70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-1" prio=10 tid=0x00007f992810c000 nid=0x77a0 waiting on
>> condition [0x00007f9915db4000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae45c728> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-2" prio=10 tid=0x00007f991000c000 nid=0x779f waiting
>> for monitor entry [0x00007f9915eb5000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-1" prio=10 tid=0x00007f991000a000 nid=0x779e waiting
>> for monitor entry [0x00007f9915fb6000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-7" prio=10 tid=0x00007f9928109800 nid=0x779d waiting on
>> condition [0x00007f99160b7000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae19d018> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-6" prio=10 tid=0x00007f98e8003800 nid=0x779c waiting on
>> condition [0x00007f99161b8000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae1ac6e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-5" prio=10 tid=0x00007f991800b800 nid=0x779b waiting on
>> condition [0x00007f99162b9000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae2367a8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-4" prio=10 tid=0x00007f9900017000 nid=0x779a waiting on
>> condition [0x00007f99163ba000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae1adfe8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-3" prio=10 tid=0x00007f9928107800 nid=0x7799 waiting on
>> condition [0x00007f99164bb000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae1af8f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-2" prio=10 tid=0x00007f98e8002000 nid=0x7798 waiting on
>> condition [0x00007f99165bc000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae231bf8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-1" prio=10 tid=0x00007f9918002800 nid=0x7797 waiting on
>> condition [0x00007f99166bd000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae233500> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-exec-0" prio=10 tid=0x00007f9900015000 nid=0x7796 waiting on
>> condition [0x00007f99167be000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae234e08> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-worker-3" prio=10 tid=0x00007f9928036800 nid=0x7795 runnable
>> [0x00007f99168bf000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae3b2e10> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae22fc08> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae3b2d20> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-worker-2" prio=10 tid=0x00007f9928035000 nid=0x7794 runnable
>> [0x00007f99169c0000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae24cd00> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae24ff20> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae24cc00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-worker-1" prio=10 tid=0x00007f9928034000 nid=0x7793 runnable
>> [0x00007f9916ac1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae19c400> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae460d48> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae19c310> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-client-worker-0" prio=10 tid=0x00007f9928039800 nid=0x7792 runnable
>> [0x00007f9916bc2000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae2d0238> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae2d0258> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae2d01f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-exec-0" prio=10 tid=0x00007f98f800a000 nid=0x7791 waiting on
>> condition [0x00007f9916cc3000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae4577a8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.LinkedBlockingQueue.poll(
>> LinkedBlockingQueue.java:467)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
>> SingleThreadEventExecutor.java:219)
>> at io.netty.util.concurrent.DefaultEventExecutor.run(
>> DefaultEventExecutor.java:34)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-worker-0" prio=10 tid=0x00007f9910009800 nid=0x7790 waiting
>> for monitor entry [0x00007f9916ec5000]
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:234)
>> - waiting to lock <0x00000000ae3c8458> (a
>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:114)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:60)
>> at org.apache.giraph.comm.netty.handler.
>> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
>> java:36)
>> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
>> channelRead(RequestServerHandler.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
>> DefaultChannelHandlerContext.java:324)
>> at org.apache.giraph.comm.netty.handler.RequestDecoder.
>> channelRead(RequestDecoder.java:103)
>> at io.netty.channel.DefaultChannelHandlerContext.
>> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>> at io.netty.channel.DefaultChannelHandlerContext.access$700(
>> DefaultChannelHandlerContext.java:29)
>> at io.netty.channel.DefaultChannelHandlerContext$8.run(
>> DefaultChannelHandlerContext.java:329)
>> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
>> SingleThreadEventExecutor.java:354)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "Thread-12" prio=10 tid=0x00007f9928a74800 nid=0x778c waiting on condition
>> [0x00007f9916dc4000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.giraph.worker.WorkerProgressWriter$1.run(
>> WorkerProgressWriter.java:55)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "netty-server-boss-0" prio=10 tid=0x00007f9928a73000 nid=0x778a runnable
>> [0x00007f9916fc6000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae3e94c8> (a io.netty.channel.nio.
>> SelectedSelectionKeySet)
>> - locked <0x00000000ae3ea7d0> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae3e93c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:101)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "main-EventThread" daemon prio=10 tid=0x00007f99287f4000 nid=0x777e
>> waiting on condition [0x00007f99170c7000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae3d6f90> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> at java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(
>> ClientCnxn.java:491)
>>
>> "main-SendThread(cloud-18.dima.tu-berlin.de:22181)" daemon prio=10
>> tid=0x00007f99287d5800 nid=0x777d runnable [0x00007f99172c9000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:81)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked <0x00000000ae3d6aa0> (a sun.nio.ch.Util$2)
>> - locked <0x00000000ae3d6a90> (a java.util.Collections$
>> UnmodifiableSet)
>> - locked <0x00000000ae3d6830> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
>> ClientCnxnSocketNIO.java:338)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(
>> ClientCnxn.java:1068)
>>
>> "LeaseChecker" daemon prio=10 tid=0x00007f99287cb800 nid=0x777c waiting on
>> condition [0x00007f99171c8000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(
>> DFSClient.java:1379)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f99287c0000
>> nid=0x777a waiting on condition [0x00007f99173ca000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae3a2fb0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1043)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1103)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f99287be000
>> nid=0x7779 waiting on condition [0x00007f99174cb000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ae3a2fb0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1043)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1103)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "communication thread" daemon prio=10 tid=0x00007f9928699800 nid=0x7771
>> waiting on condition [0x00007f99175cc000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:654)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> "Timer for 'MapTask' metrics system" daemon prio=10 tid=0x00007f9928673000
>> nid=0x776f in Object.wait() [0x00007f99179df000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000000ae393a08> (a java.util.TaskQueue)
>> at java.util.TimerThread.mainLoop(Timer.java:552)
>> - locked <0x00000000ae393a08> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "Thread for syncLogs" daemon prio=10 tid=0x00007f99284e6800 nid=0x776e
>> waiting on condition [0x00007f9917ae0000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.mapred.Child$3.run(Child.java:139)
>>
>> "IPC Client (47) connection to /127.0.0.1:36307 from
>> job_201402201052_0002" daemon prio=10 tid=0x00007f99284e0800 nid=0x776d in
>> Object.wait() [0x00007f9917be1000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$
>> Connection)
>> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.
>> java:706)
>> - locked <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$
>> Connection)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
>>
>> "Service Thread" daemon prio=10 tid=0x00007f9928105000 nid=0x776b runnable
>> [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" daemon prio=10 tid=0x00007f9928102800 nid=0x776a
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" daemon prio=10 tid=0x00007f99280ff800 nid=0x7769
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x00007f99280fd800 nid=0x7768
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x00007f99280b0800 nid=0x7767 in
>> Object.wait() [0x00007f991d50c000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$
>> Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>> - locked <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$
>> Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>>
>> "Reference Handler" daemon prio=10 tid=0x00007f99280ae800 nid=0x7766 in
>> Object.wait() [0x00007f991d60d000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
>> at java.lang.Object.wait(Object.java:503)
>> at java.lang.ref.Reference$ReferenceHandler.run(
>> Reference.java:133)
>> - locked <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
>>
>> "main" prio=10 tid=0x00007f992800b000 nid=0x7757 in Object.wait()
>> [0x00007f993111b000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000000ae3b3740> (a java.util.concurrent.
>> ConcurrentHashMap)
>> at org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(
>> NettyClient.java:710)
>> - locked <0x00000000ae3b3740> (a java.util.concurrent.
>> ConcurrentHashMap)
>> at org.apache.giraph.comm.netty.NettyClient.waitAllRequests(
>> NettyClient.java:685)
>> at org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(
>> NettyWorkerClient.java:148)
>> at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
>> BspServiceWorker.java:291)
>> at org.apache.giraph.worker.BspServiceWorker.loadVertices(
>> BspServiceWorker.java:328)
>> at org.apache.giraph.worker.BspServiceWorker.setup(
>> BspServiceWorker.java:509)
>> at org.apache.giraph.graph.GraphTaskManager.execute(
>> GraphTaskManager.java:262)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>> UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>> "VM Thread" prio=10 tid=0x00007f99280a7000 nid=0x7765 runnable
>>
>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9928018800 nid=0x7758
>> runnable
>>
>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f992801a800 nid=0x7759
>> runnable
>>
>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f992801c800 nid=0x775a
>> runnable
>>
>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f992801e800 nid=0x775b
>> runnable
>>
>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f9928020000 nid=0x775c
>> runnable
>>
>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f9928022000 nid=0x775d
>> runnable
>>
>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f9928024000 nid=0x775e
>> runnable
>>
>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f9928025800 nid=0x775f
>> runnable
>>
>> "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f9928027800 nid=0x7760
>> runnable
>>
>> "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f9928029800 nid=0x7761
>> runnable
>>
>> "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f992802b000 nid=0x7762
>> runnable
>>
>> "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f992802d000 nid=0x7763
>> runnable
>>
>> "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f992802f000 nid=0x7764
>> runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x00007f9928117800 nid=0x776c
>> waiting on condition
>>
>> JNI global references: 414
>>
>>
>>
>> On 02/16/2014 12:01 AM, Armando Miraglia wrote:
>>
>>> Hi Sebastian.
>>>
>>> I gave a quick look to the source code of DiskPartition and I realised
>>> I forgot to correctly manage the concurrent management of one variable.
>>> The patch I am sending you now should fix this problem. However, I am
>>> not sure whether this was the reason for your issue, so it is still
>>> possible that your issue is there.
>>>
>>> If you could test it, it would be great.
>>>
>>> Cheers,
>>> A.
>>>
>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
I have the feeling that an IOException is being swallowed somewhere. For
example, Sebastian, are you sure that you have enough space locally and
permissions? I'll investigate this.
On Thu, Feb 20, 2014 at 11:09 AM, Sebastian Schelter <ss...@apache.org> wrote:
> Hi Armando & Claudio,
>
> I tested your latest patch, unfortunately it still doesn't work for me.
>
> I applied the patch to the current trunk, copied my algorithm code and ran
> it on the cluster. If it runs in memory, everything works well. If I enable
> out-of-core I run into the deadlock as before.
>
> Here's the command I used:
>
> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner org.apache.giraph.examples.hyperball.HyperBall
> --vertexInputFormat org.apache.giraph.examples.hyperball.HyperBallTextInputFormat
> --vertexInputPath hdfs:///ssc/grades/data/twitter-negative/
> --vertexOutputFormat org.apache.giraph.io.formats.IdWithValueTextOutputFormat
> --outputPath hdfs:///ssc/tmp-123/ --combiner org.apache.giraph.examples.hyperball.HyperLogLogCombiner
> --outEdges org.apache.giraph.edge.LongNullArrayEdges --workers 24
> --customArguments giraph.oneToAllMsgSending=true,giraph.isStaticGraph=
> true,giraph.numComputeThreads=15,giraph.numInputThreads=15,
> giraph.numOutputThreads=15,giraph.maxNumberOfSupersteps=
> 30,giraph.useOutOfCoreGraph=true,giraph.maxPartitionsInMemory=20
>
> Best,
> Sebastian
>
> Here's a stracktrace of one workers:
>
> 2014-02-20 11:05:17
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f98f0096000 nid=0x78cb waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "netty-server-worker-15" prio=10 tid=0x00007f9910025000 nid=0x77b3
> runnable [0x00007f9914aa1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000adfc3140> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000adfc3218> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000adfc3040> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-14" prio=10 tid=0x00007f9910023000 nid=0x77b2
> runnable [0x00007f9914ba2000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae3a0680> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae22fae8> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae3a0590> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-13" prio=10 tid=0x00007f9910021000 nid=0x77b1
> runnable [0x00007f9914ca3000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae474e30> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae4780c8> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae474d40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-12" prio=10 tid=0x00007f991001f800 nid=0x77b0
> runnable [0x00007f9914da4000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000adfee950> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000adfeec20> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000adfee860> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-11" prio=10 tid=0x00007f991001d800 nid=0x77af
> runnable [0x00007f9914ea5000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae0fc9b0> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae0fcc80> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae0fc8c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-10" prio=10 tid=0x00007f991001c000 nid=0x77ae waiting
> for monitor entry [0x00007f9914fa6000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-9" prio=10 tid=0x00007f991001a800 nid=0x77ad runnable
> [0x00007f99150a7000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000adf80e78> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000adf81148> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000adf80d88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-7" prio=10 tid=0x00007f98f8015800 nid=0x77ac waiting on
> condition [0x00007f99151a8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae444d38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-8" prio=10 tid=0x00007f9910018800 nid=0x77ab runnable
> [0x00007f99152a9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae0fd200> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae0fd2d8> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae0fd100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-7" prio=10 tid=0x00007f9910016800 nid=0x77aa waiting
> for monitor entry [0x00007f99153aa000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-6" prio=10 tid=0x00007f9928452800 nid=0x77a9 waiting on
> condition [0x00007f99154ab000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae4466a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-5" prio=10 tid=0x00007f99006d6000 nid=0x77a8 waiting on
> condition [0x00007f99155ac000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae44df48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-6" prio=10 tid=0x00007f9910014800 nid=0x77a7 waiting
> for monitor entry [0x00007f99156ad000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-4" prio=10 tid=0x00007f9908661800 nid=0x77a6 waiting on
> condition [0x00007f99157ae000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae4287a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-3" prio=10 tid=0x00007f98f0005800 nid=0x77a5 waiting on
> condition [0x00007f99158af000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae452518> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-5" prio=10 tid=0x00007f9910012800 nid=0x77a4 waiting
> for monitor entry [0x00007f99159b0000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-4" prio=10 tid=0x00007f9910010800 nid=0x77a3 runnable
> [0x00007f9915ab1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000adfadba0> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000adfb86d8> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000adfadab0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-3" prio=10 tid=0x00007f991000e800 nid=0x77a2 runnable
> [0x00007f9915bb1000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:242)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00000005be8e5778> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at org.apache.giraph.edge.LongNullArrayEdges.readFields(
> LongNullArrayEdges.java:161)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> readOutEdges(DiskBackedPartitionStore.java:387)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> loadPartition(DiskBackedPartitionStore.java:441)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getPartition(DiskBackedPartitionStore.java:771)
> - locked <0x00000000ae3c8458> (a org.apache.giraph.partition.
> DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:238)
> - locked <0x00000000ae3c8458> (a org.apache.giraph.partition.
> DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-2" prio=10 tid=0x0000000000eb2800 nid=0x77a1 waiting on
> condition [0x00007f9915cb3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae453e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-1" prio=10 tid=0x00007f992810c000 nid=0x77a0 waiting on
> condition [0x00007f9915db4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae45c728> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-2" prio=10 tid=0x00007f991000c000 nid=0x779f waiting
> for monitor entry [0x00007f9915eb5000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-1" prio=10 tid=0x00007f991000a000 nid=0x779e waiting
> for monitor entry [0x00007f9915fb6000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-7" prio=10 tid=0x00007f9928109800 nid=0x779d waiting on
> condition [0x00007f99160b7000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae19d018> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-6" prio=10 tid=0x00007f98e8003800 nid=0x779c waiting on
> condition [0x00007f99161b8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae1ac6e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-5" prio=10 tid=0x00007f991800b800 nid=0x779b waiting on
> condition [0x00007f99162b9000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae2367a8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-4" prio=10 tid=0x00007f9900017000 nid=0x779a waiting on
> condition [0x00007f99163ba000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae1adfe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-3" prio=10 tid=0x00007f9928107800 nid=0x7799 waiting on
> condition [0x00007f99164bb000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae1af8f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-2" prio=10 tid=0x00007f98e8002000 nid=0x7798 waiting on
> condition [0x00007f99165bc000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae231bf8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-1" prio=10 tid=0x00007f9918002800 nid=0x7797 waiting on
> condition [0x00007f99166bd000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae233500> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-exec-0" prio=10 tid=0x00007f9900015000 nid=0x7796 waiting on
> condition [0x00007f99167be000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae234e08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-worker-3" prio=10 tid=0x00007f9928036800 nid=0x7795 runnable
> [0x00007f99168bf000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae3b2e10> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae22fc08> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae3b2d20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-worker-2" prio=10 tid=0x00007f9928035000 nid=0x7794 runnable
> [0x00007f99169c0000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae24cd00> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae24ff20> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae24cc00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-worker-1" prio=10 tid=0x00007f9928034000 nid=0x7793 runnable
> [0x00007f9916ac1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae19c400> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae460d48> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae19c310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-client-worker-0" prio=10 tid=0x00007f9928039800 nid=0x7792 runnable
> [0x00007f9916bc2000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae2d0238> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae2d0258> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae2d01f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-exec-0" prio=10 tid=0x00007f98f800a000 nid=0x7791 waiting on
> condition [0x00007f9916cc3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae4577a8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(
> LinkedBlockingQueue.java:467)
> at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(
> SingleThreadEventExecutor.java:219)
> at io.netty.util.concurrent.DefaultEventExecutor.run(
> DefaultEventExecutor.java:34)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-worker-0" prio=10 tid=0x00007f9910009800 nid=0x7790 waiting
> for monitor entry [0x00007f9916ec5000]
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:234)
> - waiting to lock <0x00000000ae3c8458> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:114)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:60)
> at org.apache.giraph.comm.netty.handler.
> WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.
> java:36)
> at org.apache.giraph.comm.netty.handler.RequestServerHandler.
> channelRead(RequestServerHandler.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.handler.RequestDecoder.
> channelRead(RequestDecoder.java:103)
> at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.access$700(
> DefaultChannelHandlerContext.java:29)
> at io.netty.channel.DefaultChannelHandlerContext$8.run(
> DefaultChannelHandlerContext.java:329)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(
> SingleThreadEventExecutor.java:354)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "Thread-12" prio=10 tid=0x00007f9928a74800 nid=0x778c waiting on condition
> [0x00007f9916dc4000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.giraph.worker.WorkerProgressWriter$1.run(
> WorkerProgressWriter.java:55)
> at java.lang.Thread.run(Thread.java:722)
>
> "netty-server-boss-0" prio=10 tid=0x00007f9928a73000 nid=0x778a runnable
> [0x00007f9916fc6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae3e94c8> (a io.netty.channel.nio.
> SelectedSelectionKeySet)
> - locked <0x00000000ae3ea7d0> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae3e93c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:722)
>
> "main-EventThread" daemon prio=10 tid=0x00007f99287f4000 nid=0x777e
> waiting on condition [0x00007f99170c7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae3d6f90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(
> ClientCnxn.java:491)
>
> "main-SendThread(cloud-18.dima.tu-berlin.de:22181)" daemon prio=10
> tid=0x00007f99287d5800 nid=0x777d runnable [0x00007f99172c9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
> java:81)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000ae3d6aa0> (a sun.nio.ch.Util$2)
> - locked <0x00000000ae3d6a90> (a java.util.Collections$
> UnmodifiableSet)
> - locked <0x00000000ae3d6830> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:338)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(
> ClientCnxn.java:1068)
>
> "LeaseChecker" daemon prio=10 tid=0x00007f99287cb800 nid=0x777c waiting on
> condition [0x00007f99171c8000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(
> DFSClient.java:1379)
> at java.lang.Thread.run(Thread.java:722)
>
> "metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f99287c0000
> nid=0x777a waiting on condition [0x00007f99173ca000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae3a2fb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1043)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1103)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
>
> "metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f99287be000
> nid=0x7779 waiting on condition [0x00007f99174cb000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ae3a2fb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1043)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1103)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
>
> "communication thread" daemon prio=10 tid=0x00007f9928699800 nid=0x7771
> waiting on condition [0x00007f99175cc000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:654)
> at java.lang.Thread.run(Thread.java:722)
>
> "Timer for 'MapTask' metrics system" daemon prio=10 tid=0x00007f9928673000
> nid=0x776f in Object.wait() [0x00007f99179df000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000ae393a08> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000ae393a08> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f99284e6800 nid=0x776e
> waiting on condition [0x00007f9917ae0000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$3.run(Child.java:139)
>
> "IPC Client (47) connection to /127.0.0.1:36307 from
> job_201402201052_0002" daemon prio=10 tid=0x00007f99284e0800 nid=0x776d in
> Object.wait() [0x00007f9917be1000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$
> Connection)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.
> java:706)
> - locked <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$
> Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
>
> "Service Thread" daemon prio=10 tid=0x00007f9928105000 nid=0x776b runnable
> [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f9928102800 nid=0x776a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f99280ff800 nid=0x7769
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f99280fd800 nid=0x7768
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f99280b0800 nid=0x7767 in
> Object.wait() [0x00007f991d50c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$
> Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$
> Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>
> "Reference Handler" daemon prio=10 tid=0x00007f99280ae800 nid=0x7766 in
> Object.wait() [0x00007f991d60d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(
> Reference.java:133)
> - locked <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x00007f992800b000 nid=0x7757 in Object.wait()
> [0x00007f993111b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000ae3b3740> (a java.util.concurrent.
> ConcurrentHashMap)
> at org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(
> NettyClient.java:710)
> - locked <0x00000000ae3b3740> (a java.util.concurrent.
> ConcurrentHashMap)
> at org.apache.giraph.comm.netty.NettyClient.waitAllRequests(
> NettyClient.java:685)
> at org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(
> NettyWorkerClient.java:148)
> at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
> BspServiceWorker.java:291)
> at org.apache.giraph.worker.BspServiceWorker.loadVertices(
> BspServiceWorker.java:328)
> at org.apache.giraph.worker.BspServiceWorker.setup(
> BspServiceWorker.java:509)
> at org.apache.giraph.graph.GraphTaskManager.execute(
> GraphTaskManager.java:262)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> "VM Thread" prio=10 tid=0x00007f99280a7000 nid=0x7765 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9928018800 nid=0x7758
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f992801a800 nid=0x7759
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f992801c800 nid=0x775a
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f992801e800 nid=0x775b
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f9928020000 nid=0x775c
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f9928022000 nid=0x775d
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f9928024000 nid=0x775e
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f9928025800 nid=0x775f
> runnable
>
> "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f9928027800 nid=0x7760
> runnable
>
> "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f9928029800 nid=0x7761
> runnable
>
> "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f992802b000 nid=0x7762
> runnable
>
> "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f992802d000 nid=0x7763
> runnable
>
> "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f992802f000 nid=0x7764
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f9928117800 nid=0x776c
> waiting on condition
>
> JNI global references: 414
>
>
>
> On 02/16/2014 12:01 AM, Armando Miraglia wrote:
>
>> Hi Sebastian.
>>
>> I gave a quick look to the source code of DiskPartition and I realised
>> I forgot to correctly manage the concurrent management of one variable.
>> The patch I am sending you now should fix this problem. However, I am
>> not sure whether this was the reason for your issue, so it is still
>> possible that your issue is there.
>>
>> If you could test it, it would be great.
>>
>> Cheers,
>> A.
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Armando & Claudio,
I tested your latest patch, unfortunately it still doesn't work for me.
I applied the patch to the current trunk, copied my algorithm code and
ran it on the cluster. If it runs in memory, everything works well. If I
enable out-of-core I run into the deadlock as before.
Here's the command I used:
hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.hyperball.HyperBall --vertexInputFormat
org.apache.giraph.examples.hyperball.HyperBallTextInputFormat
--vertexInputPath hdfs:///ssc/grades/data/twitter-negative/
--vertexOutputFormat
org.apache.giraph.io.formats.IdWithValueTextOutputFormat --outputPath
hdfs:///ssc/tmp-123/ --combiner
org.apache.giraph.examples.hyperball.HyperLogLogCombiner --outEdges
org.apache.giraph.edge.LongNullArrayEdges --workers 24 --customArguments
giraph.oneToAllMsgSending=true,giraph.isStaticGraph=true,giraph.numComputeThreads=15,giraph.numInputThreads=15,giraph.numOutputThreads=15,giraph.maxNumberOfSupersteps=30,giraph.useOutOfCoreGraph=true,giraph.maxPartitionsInMemory=20
Best,
Sebastian
Here's a stracktrace of one workers:
2014-02-20 11:05:17
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f98f0096000 nid=0x78cb
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"netty-server-worker-15" prio=10 tid=0x00007f9910025000 nid=0x77b3
runnable [0x00007f9914aa1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000adfc3140> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000adfc3218> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000adfc3040> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-14" prio=10 tid=0x00007f9910023000 nid=0x77b2
runnable [0x00007f9914ba2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae3a0680> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae22fae8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae3a0590> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-13" prio=10 tid=0x00007f9910021000 nid=0x77b1
runnable [0x00007f9914ca3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae474e30> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae4780c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae474d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-12" prio=10 tid=0x00007f991001f800 nid=0x77b0
runnable [0x00007f9914da4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000adfee950> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000adfeec20> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000adfee860> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-11" prio=10 tid=0x00007f991001d800 nid=0x77af
runnable [0x00007f9914ea5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae0fc9b0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae0fcc80> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae0fc8c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-10" prio=10 tid=0x00007f991001c000 nid=0x77ae
waiting for monitor entry [0x00007f9914fa6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-9" prio=10 tid=0x00007f991001a800 nid=0x77ad
runnable [0x00007f99150a7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000adf80e78> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000adf81148> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000adf80d88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-7" prio=10 tid=0x00007f98f8015800 nid=0x77ac waiting
on condition [0x00007f99151a8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae444d38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-8" prio=10 tid=0x00007f9910018800 nid=0x77ab
runnable [0x00007f99152a9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae0fd200> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae0fd2d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae0fd100> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-7" prio=10 tid=0x00007f9910016800 nid=0x77aa
waiting for monitor entry [0x00007f99153aa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-6" prio=10 tid=0x00007f9928452800 nid=0x77a9 waiting
on condition [0x00007f99154ab000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae4466a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-5" prio=10 tid=0x00007f99006d6000 nid=0x77a8 waiting
on condition [0x00007f99155ac000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae44df48> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-6" prio=10 tid=0x00007f9910014800 nid=0x77a7
waiting for monitor entry [0x00007f99156ad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-4" prio=10 tid=0x00007f9908661800 nid=0x77a6 waiting
on condition [0x00007f99157ae000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae4287a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-3" prio=10 tid=0x00007f98f0005800 nid=0x77a5 waiting
on condition [0x00007f99158af000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae452518> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-5" prio=10 tid=0x00007f9910012800 nid=0x77a4
waiting for monitor entry [0x00007f99159b0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-4" prio=10 tid=0x00007f9910010800 nid=0x77a3
runnable [0x00007f9915ab1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000adfadba0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000adfb86d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000adfadab0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-3" prio=10 tid=0x00007f991000e800 nid=0x77a2
runnable [0x00007f9915bb1000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:242)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000005be8e5778> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.giraph.edge.LongNullArrayEdges.readFields(LongNullArrayEdges.java:161)
at
org.apache.giraph.partition.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:387)
at
org.apache.giraph.partition.DiskBackedPartitionStore.loadPartition(DiskBackedPartitionStore.java:441)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getPartition(DiskBackedPartitionStore.java:771)
- locked <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:238)
- locked <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-2" prio=10 tid=0x0000000000eb2800 nid=0x77a1 waiting
on condition [0x00007f9915cb3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae453e70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-1" prio=10 tid=0x00007f992810c000 nid=0x77a0 waiting
on condition [0x00007f9915db4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae45c728> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-2" prio=10 tid=0x00007f991000c000 nid=0x779f
waiting for monitor entry [0x00007f9915eb5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-1" prio=10 tid=0x00007f991000a000 nid=0x779e
waiting for monitor entry [0x00007f9915fb6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-7" prio=10 tid=0x00007f9928109800 nid=0x779d waiting
on condition [0x00007f99160b7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae19d018> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-6" prio=10 tid=0x00007f98e8003800 nid=0x779c waiting
on condition [0x00007f99161b8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae1ac6e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-5" prio=10 tid=0x00007f991800b800 nid=0x779b waiting
on condition [0x00007f99162b9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae2367a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-4" prio=10 tid=0x00007f9900017000 nid=0x779a waiting
on condition [0x00007f99163ba000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae1adfe8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-3" prio=10 tid=0x00007f9928107800 nid=0x7799 waiting
on condition [0x00007f99164bb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae1af8f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-2" prio=10 tid=0x00007f98e8002000 nid=0x7798 waiting
on condition [0x00007f99165bc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae231bf8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-1" prio=10 tid=0x00007f9918002800 nid=0x7797 waiting
on condition [0x00007f99166bd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae233500> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-exec-0" prio=10 tid=0x00007f9900015000 nid=0x7796 waiting
on condition [0x00007f99167be000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae234e08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-worker-3" prio=10 tid=0x00007f9928036800 nid=0x7795
runnable [0x00007f99168bf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae3b2e10> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae22fc08> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae3b2d20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-worker-2" prio=10 tid=0x00007f9928035000 nid=0x7794
runnable [0x00007f99169c0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae24cd00> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae24ff20> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae24cc00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-worker-1" prio=10 tid=0x00007f9928034000 nid=0x7793
runnable [0x00007f9916ac1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae19c400> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae460d48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae19c310> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-client-worker-0" prio=10 tid=0x00007f9928039800 nid=0x7792
runnable [0x00007f9916bc2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae2d0238> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae2d0258> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae2d01f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-exec-0" prio=10 tid=0x00007f98f800a000 nid=0x7791 waiting
on condition [0x00007f9916cc3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae4577a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
at
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"netty-server-worker-0" prio=10 tid=0x00007f9910009800 nid=0x7790
waiting for monitor entry [0x00007f9916ec5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
- waiting to lock <0x00000000ae3c8458> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
at
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
at
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"Thread-12" prio=10 tid=0x00007f9928a74800 nid=0x778c waiting on
condition [0x00007f9916dc4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.giraph.worker.WorkerProgressWriter$1.run(WorkerProgressWriter.java:55)
at java.lang.Thread.run(Thread.java:722)
"netty-server-boss-0" prio=10 tid=0x00007f9928a73000 nid=0x778a runnable
[0x00007f9916fc6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae3e94c8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000ae3ea7d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae3e93c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:722)
"main-EventThread" daemon prio=10 tid=0x00007f99287f4000 nid=0x777e
waiting on condition [0x00007f99170c7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae3d6f90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"main-SendThread(cloud-18.dima.tu-berlin.de:22181)" daemon prio=10
tid=0x00007f99287d5800 nid=0x777d runnable [0x00007f99172c9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ae3d6aa0> (a sun.nio.ch.Util$2)
- locked <0x00000000ae3d6a90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ae3d6830> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"LeaseChecker" daemon prio=10 tid=0x00007f99287cb800 nid=0x777c waiting
on condition [0x00007f99171c8000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
at java.lang.Thread.run(Thread.java:722)
"metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f99287c0000
nid=0x777a waiting on condition [0x00007f99173ca000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae3a2fb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f99287be000
nid=0x7779 waiting on condition [0x00007f99174cb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae3a2fb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"communication thread" daemon prio=10 tid=0x00007f9928699800 nid=0x7771
waiting on condition [0x00007f99175cc000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:654)
at java.lang.Thread.run(Thread.java:722)
"Timer for 'MapTask' metrics system" daemon prio=10
tid=0x00007f9928673000 nid=0x776f in Object.wait() [0x00007f99179df000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ae393a08> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000ae393a08> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Thread for syncLogs" daemon prio=10 tid=0x00007f99284e6800 nid=0x776e
waiting on condition [0x00007f9917ae0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.Child$3.run(Child.java:139)
"IPC Client (47) connection to /127.0.0.1:36307 from
job_201402201052_0002" daemon prio=10 tid=0x00007f99284e0800 nid=0x776d
in Object.wait() [0x00007f9917be1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ae325778> (a
org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
- locked <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
"Service Thread" daemon prio=10 tid=0x00007f9928105000 nid=0x776b
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f9928102800 nid=0x776a
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f99280ff800 nid=0x7769
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f99280fd800 nid=0x7768
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f99280b0800 nid=0x7767 in
Object.wait() [0x00007f991d50c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x00007f99280ae800 nid=0x7766 in
Object.wait() [0x00007f991d60d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f992800b000 nid=0x7757 in Object.wait()
[0x00007f993111b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ae3b3740> (a
java.util.concurrent.ConcurrentHashMap)
at
org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(NettyClient.java:710)
- locked <0x00000000ae3b3740> (a java.util.concurrent.ConcurrentHashMap)
at
org.apache.giraph.comm.netty.NettyClient.waitAllRequests(NettyClient.java:685)
at
org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(NettyWorkerClient.java:148)
at
org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:291)
at
org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:328)
at
org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:509)
at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:262)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
"VM Thread" prio=10 tid=0x00007f99280a7000 nid=0x7765 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9928018800
nid=0x7758 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f992801a800
nid=0x7759 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f992801c800
nid=0x775a runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f992801e800
nid=0x775b runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f9928020000
nid=0x775c runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f9928022000
nid=0x775d runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f9928024000
nid=0x775e runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f9928025800
nid=0x775f runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f9928027800
nid=0x7760 runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f9928029800
nid=0x7761 runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f992802b000
nid=0x7762 runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f992802d000
nid=0x7763 runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f992802f000
nid=0x7764 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f9928117800 nid=0x776c
waiting on condition
JNI global references: 414
On 02/16/2014 12:01 AM, Armando Miraglia wrote:
> Hi Sebastian.
>
> I gave a quick look to the source code of DiskPartition and I realised
> I forgot to correctly manage the concurrent management of one variable.
> The patch I am sending you now should fix this problem. However, I am
> not sure whether this was the reason for your issue, so it is still
> possible that your issue is there.
>
> If you could test it, it would be great.
>
> Cheers,
> A.
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
On Thu, Feb 13, 2014 at 06:05:53PM +0100, Sebastian Schelter wrote:
> I'm using 24 workers to process a dataset of 50M vertices. Where can I see
> the number of partitions assigned to each worker?
Firstly, The number of partitions assigned to each worker can be seen in
the worker logs (you can access them via the task page). It is
sufficient you look for "partition" in the page, and you'll find a log
entry providing the information (is logged by the ProgressUtils class).
Secondly, can you please provide all the options you are setting for the
computation? In case you are setting the options in the giraph-site.xml
file, please provide those information as well. In this manner I can try
to replicate the issue, since all the tests I have been doing had no
issues.
Cheers,
Armando
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
can you try with these options?
hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar \
org.apache.giraph.GiraphRunner \
-D giraph.oneToAllMsgSending=true \
-D giraph.isStaticGraph=true \
-D giraph.numComputeThreads=15 \
-D giraph.numInputThreads=15 \
-D giraph.numOutputThreads=15 \
-D giraph.maxNumberOfSupersteps=30 \
-D giraph.useOutOfCoreGraph=true \
-D giraph.maxPartitionsInMemory=20 \
-D giraph.userPartitionCount=24 \
org.apache.giraph.examples.hyperball.HyperBall \
--vertexInputFormat org.apache.giraph.examples.hyperball.HyperBallTextInputFormat \
--vertexInputPath input/ \
--vertexOutputFormat org.apache.giraph.io.formats.IdWithValueTextOutputFormat \
--outputPath output/hyperball \
--combiner org.apache.giraph.comm.messages.HyperLogLogCombiner \
--outEdges org.apache.giraph.edge.LongNullArrayEdges \
--workers 1
so single worker, but the same number of partitions.
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
I ran the job without the caching trick in the inputformat and still run
into the freeze.
On 02/15/2014 08:52 PM, Claudio Martella wrote:
> I don't know, maybe I'm missing something, or there's a bug there as well.
> I do agree that this is spooky. Armando has tested it also with the
> WattsStrogatzInputformat, that creates another type of graph. For what I
> understand, this should not happen due to the topology. I think we should
> just try to replicate this behavior, hopefully without a very large graph
> that makes debugging difficult.
>
>
> On Sat, Feb 15, 2014 at 8:42 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> I copied the caching from o.a.g.io.formats.IntIntNullTextInputFormat and
>> it worked well during my tests (it did not happen that all vertices had the
>> same id).
>>
>> I'm happy to remove this and rerun the tests. It's strange that
>> out-of-core works with PageRank on a generated graph, but not with
>> Hyperball on the twitter graph. The generated graph has a uniform degree
>> distribution, while the twitter graph's degree distribution is heavily
>> skewed, can that have an influence on the behavior of ooc?
>>
>> Best,
>> Sebastian
>>
>>
>>
>> On 02/15/2014 08:32 PM, Claudio Martella wrote:
>>
>>> Sebastian, I had a look at your vertexinputformat. I think there might be
>>> a
>>> bug. Why are you caching/reusing the id? This way every vertex parsed by
>>> the vertexreader will share the same ID object, and hence have the same
>>> ID.
>>> I think this is broken. you should instantiate a new ID object in the
>>> preprocessLine.
>>> Can you try like that?
>>>
>>>
>>> On Thu, Feb 13, 2014 at 9:50 PM, Sebastian Schelter <ss...@apache.org>
>>> wrote:
>>>
>>> Hi Armando,
>>>>
>>>> I uploaded my test code to github at:
>>>>
>>>> https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
>>>>
>>>> I'm working on an algorithm to estimate the neighborhood function of the
>>>> graph (similar to [1]). I'm running this on the transposed adjacency
>>>> matrix
>>>> of a snapshot of the twitter follower graph [2]. For this graph
>>>> out-of-core
>>>> is not necessary, but I would like to run my algorithm on another larger
>>>> graph that doesn't fit into the aggregated main memory of the cluster
>>>> anymore.
>>>>
>>>> I think for testing purposes, you can run it on any large graph in
>>>> adjacency form.
>>>>
>>>> Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks
>>>> per
>>>> machine. I use the following options to run the algorithm:
>>>>
>>>> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-
>>>> with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>>
>>>> org.apache.giraph.examples.hyperball.HyperBall
>>>>
>>>> --vertexInputFormat org.apache.giraph.examples.hyperball.
>>>> HyperBallTextInputFormat
>>>>
>>>> --vertexInputPath hdfs:///ssc/twitter-negative/
>>>>
>>>> --vertexOutputFormat org.apache.giraph.io.formats.
>>>> IdWithValueTextOutputFormat
>>>>
>>>> --outputPath hdfs:///ssc/tmp-123/
>>>>
>>>> --combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
>>>>
>>>> --outEdges org.apache.giraph.edge.LongNullArrayEdges
>>>>
>>>> --workers 24
>>>>
>>>> --customArguments
>>>>
>>>> giraph.oneToAllMsgSending=true,
>>>> giraph.isStaticGraph=true,
>>>> giraph.numComputeThreads=15,
>>>> giraph.numInputThreads=15,
>>>> giraph.numOutputThreads=15,
>>>> giraph.maxNumberOfSupersteps=30,
>>>> giraph.useOutOfCoreGraph=true,
>>>> giraph.maxPartitionsInMemory=20
>>>>
>>>> Best,
>>>> Sebastian
>>>>
>>>> [1] http://arxiv.org/abs/1308.2144
>>>> [2] http://konect.uni-koblenz.de/networks/twitter_mpi
>>>>
>>>>
>>>> On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>>>>
>>>>
>>>>> Hi Sebastian,
>>>>>
>>>>> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>>>>>
>>>>> No. Should I have done that?
>>>>>>
>>>>>>
>>>>> could you please provide me with the test you have done together with
>>>>> the variables that you have set during for the computation? This would
>>>>> help me a lot.
>>>>>
>>>>> Cheers,
>>>>> Armando
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
I don't know, maybe I'm missing something, or there's a bug there as well.
I do agree that this is spooky. Armando has tested it also with the
WattsStrogatzInputformat, that creates another type of graph. For what I
understand, this should not happen due to the topology. I think we should
just try to replicate this behavior, hopefully without a very large graph
that makes debugging difficult.
On Sat, Feb 15, 2014 at 8:42 PM, Sebastian Schelter <ss...@apache.org> wrote:
> I copied the caching from o.a.g.io.formats.IntIntNullTextInputFormat and
> it worked well during my tests (it did not happen that all vertices had the
> same id).
>
> I'm happy to remove this and rerun the tests. It's strange that
> out-of-core works with PageRank on a generated graph, but not with
> Hyperball on the twitter graph. The generated graph has a uniform degree
> distribution, while the twitter graph's degree distribution is heavily
> skewed, can that have an influence on the behavior of ooc?
>
> Best,
> Sebastian
>
>
>
> On 02/15/2014 08:32 PM, Claudio Martella wrote:
>
>> Sebastian, I had a look at your vertexinputformat. I think there might be
>> a
>> bug. Why are you caching/reusing the id? This way every vertex parsed by
>> the vertexreader will share the same ID object, and hence have the same
>> ID.
>> I think this is broken. you should instantiate a new ID object in the
>> preprocessLine.
>> Can you try like that?
>>
>>
>> On Thu, Feb 13, 2014 at 9:50 PM, Sebastian Schelter <ss...@apache.org>
>> wrote:
>>
>> Hi Armando,
>>>
>>> I uploaded my test code to github at:
>>>
>>> https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
>>>
>>> I'm working on an algorithm to estimate the neighborhood function of the
>>> graph (similar to [1]). I'm running this on the transposed adjacency
>>> matrix
>>> of a snapshot of the twitter follower graph [2]. For this graph
>>> out-of-core
>>> is not necessary, but I would like to run my algorithm on another larger
>>> graph that doesn't fit into the aggregated main memory of the cluster
>>> anymore.
>>>
>>> I think for testing purposes, you can run it on any large graph in
>>> adjacency form.
>>>
>>> Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks
>>> per
>>> machine. I use the following options to run the algorithm:
>>>
>>> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-
>>> with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>>
>>> org.apache.giraph.examples.hyperball.HyperBall
>>>
>>> --vertexInputFormat org.apache.giraph.examples.hyperball.
>>> HyperBallTextInputFormat
>>>
>>> --vertexInputPath hdfs:///ssc/twitter-negative/
>>>
>>> --vertexOutputFormat org.apache.giraph.io.formats.
>>> IdWithValueTextOutputFormat
>>>
>>> --outputPath hdfs:///ssc/tmp-123/
>>>
>>> --combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
>>>
>>> --outEdges org.apache.giraph.edge.LongNullArrayEdges
>>>
>>> --workers 24
>>>
>>> --customArguments
>>>
>>> giraph.oneToAllMsgSending=true,
>>> giraph.isStaticGraph=true,
>>> giraph.numComputeThreads=15,
>>> giraph.numInputThreads=15,
>>> giraph.numOutputThreads=15,
>>> giraph.maxNumberOfSupersteps=30,
>>> giraph.useOutOfCoreGraph=true,
>>> giraph.maxPartitionsInMemory=20
>>>
>>> Best,
>>> Sebastian
>>>
>>> [1] http://arxiv.org/abs/1308.2144
>>> [2] http://konect.uni-koblenz.de/networks/twitter_mpi
>>>
>>>
>>> On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>>>
>>>
>>>> Hi Sebastian,
>>>>
>>>> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>>>>
>>>> No. Should I have done that?
>>>>>
>>>>>
>>>> could you please provide me with the test you have done together with
>>>> the variables that you have set during for the computation? This would
>>>> help me a lot.
>>>>
>>>> Cheers,
>>>> Armando
>>>>
>>>>
>>>>
>>>
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
I copied the caching from o.a.g.io.formats.IntIntNullTextInputFormat and
it worked well during my tests (it did not happen that all vertices had
the same id).
I'm happy to remove this and rerun the tests. It's strange that
out-of-core works with PageRank on a generated graph, but not with
Hyperball on the twitter graph. The generated graph has a uniform degree
distribution, while the twitter graph's degree distribution is heavily
skewed, can that have an influence on the behavior of ooc?
Best,
Sebastian
On 02/15/2014 08:32 PM, Claudio Martella wrote:
> Sebastian, I had a look at your vertexinputformat. I think there might be a
> bug. Why are you caching/reusing the id? This way every vertex parsed by
> the vertexreader will share the same ID object, and hence have the same ID.
> I think this is broken. you should instantiate a new ID object in the
> preprocessLine.
> Can you try like that?
>
>
> On Thu, Feb 13, 2014 at 9:50 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> Hi Armando,
>>
>> I uploaded my test code to github at:
>>
>> https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
>>
>> I'm working on an algorithm to estimate the neighborhood function of the
>> graph (similar to [1]). I'm running this on the transposed adjacency matrix
>> of a snapshot of the twitter follower graph [2]. For this graph out-of-core
>> is not necessary, but I would like to run my algorithm on another larger
>> graph that doesn't fit into the aggregated main memory of the cluster
>> anymore.
>>
>> I think for testing purposes, you can run it on any large graph in
>> adjacency form.
>>
>> Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks per
>> machine. I use the following options to run the algorithm:
>>
>> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.hyperball.HyperBall
>>
>> --vertexInputFormat org.apache.giraph.examples.hyperball.
>> HyperBallTextInputFormat
>>
>> --vertexInputPath hdfs:///ssc/twitter-negative/
>>
>> --vertexOutputFormat org.apache.giraph.io.formats.
>> IdWithValueTextOutputFormat
>>
>> --outputPath hdfs:///ssc/tmp-123/
>>
>> --combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
>>
>> --outEdges org.apache.giraph.edge.LongNullArrayEdges
>>
>> --workers 24
>>
>> --customArguments
>>
>> giraph.oneToAllMsgSending=true,
>> giraph.isStaticGraph=true,
>> giraph.numComputeThreads=15,
>> giraph.numInputThreads=15,
>> giraph.numOutputThreads=15,
>> giraph.maxNumberOfSupersteps=30,
>> giraph.useOutOfCoreGraph=true,
>> giraph.maxPartitionsInMemory=20
>>
>> Best,
>> Sebastian
>>
>> [1] http://arxiv.org/abs/1308.2144
>> [2] http://konect.uni-koblenz.de/networks/twitter_mpi
>>
>>
>> On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>>
>>>
>>> Hi Sebastian,
>>>
>>> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>>>
>>>> No. Should I have done that?
>>>>
>>>
>>> could you please provide me with the test you have done together with
>>> the variables that you have set during for the computation? This would
>>> help me a lot.
>>>
>>> Cheers,
>>> Armando
>>>
>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
Sebastian, I had a look at your vertexinputformat. I think there might be a
bug. Why are you caching/reusing the id? This way every vertex parsed by
the vertexreader will share the same ID object, and hence have the same ID.
I think this is broken. you should instantiate a new ID object in the
preprocessLine.
Can you try like that?
On Thu, Feb 13, 2014 at 9:50 PM, Sebastian Schelter <ss...@apache.org> wrote:
> Hi Armando,
>
> I uploaded my test code to github at:
>
> https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
>
> I'm working on an algorithm to estimate the neighborhood function of the
> graph (similar to [1]). I'm running this on the transposed adjacency matrix
> of a snapshot of the twitter follower graph [2]. For this graph out-of-core
> is not necessary, but I would like to run my algorithm on another larger
> graph that doesn't fit into the aggregated main memory of the cluster
> anymore.
>
> I think for testing purposes, you can run it on any large graph in
> adjacency form.
>
> Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks per
> machine. I use the following options to run the algorithm:
>
> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
>
> org.apache.giraph.examples.hyperball.HyperBall
>
> --vertexInputFormat org.apache.giraph.examples.hyperball.
> HyperBallTextInputFormat
>
> --vertexInputPath hdfs:///ssc/twitter-negative/
>
> --vertexOutputFormat org.apache.giraph.io.formats.
> IdWithValueTextOutputFormat
>
> --outputPath hdfs:///ssc/tmp-123/
>
> --combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
>
> --outEdges org.apache.giraph.edge.LongNullArrayEdges
>
> --workers 24
>
> --customArguments
>
> giraph.oneToAllMsgSending=true,
> giraph.isStaticGraph=true,
> giraph.numComputeThreads=15,
> giraph.numInputThreads=15,
> giraph.numOutputThreads=15,
> giraph.maxNumberOfSupersteps=30,
> giraph.useOutOfCoreGraph=true,
> giraph.maxPartitionsInMemory=20
>
> Best,
> Sebastian
>
> [1] http://arxiv.org/abs/1308.2144
> [2] http://konect.uni-koblenz.de/networks/twitter_mpi
>
>
> On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>
>>
>> Hi Sebastian,
>>
>> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>>
>>> No. Should I have done that?
>>>
>>
>> could you please provide me with the test you have done together with
>> the variables that you have set during for the computation? This would
>> help me a lot.
>>
>> Cheers,
>> Armando
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Armando,
I uploaded my test code to github at:
https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
I'm working on an algorithm to estimate the neighborhood function of the
graph (similar to [1]). I'm running this on the transposed adjacency
matrix of a snapshot of the twitter follower graph [2]. For this graph
out-of-core is not necessary, but I would like to run my algorithm on
another larger graph that doesn't fit into the aggregated main memory of
the cluster anymore.
I think for testing purposes, you can run it on any large graph in
adjacency form.
Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks
per machine. I use the following options to run the algorithm:
hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.hyperball.HyperBall
--vertexInputFormat
org.apache.giraph.examples.hyperball.HyperBallTextInputFormat
--vertexInputPath hdfs:///ssc/twitter-negative/
--vertexOutputFormat
org.apache.giraph.io.formats.IdWithValueTextOutputFormat
--outputPath hdfs:///ssc/tmp-123/
--combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
--outEdges org.apache.giraph.edge.LongNullArrayEdges
--workers 24
--customArguments
giraph.oneToAllMsgSending=true,
giraph.isStaticGraph=true,
giraph.numComputeThreads=15,
giraph.numInputThreads=15,
giraph.numOutputThreads=15,
giraph.maxNumberOfSupersteps=30,
giraph.useOutOfCoreGraph=true,
giraph.maxPartitionsInMemory=20
Best,
Sebastian
[1] http://arxiv.org/abs/1308.2144
[2] http://konect.uni-koblenz.de/networks/twitter_mpi
On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>
> Hi Sebastian,
>
> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>> No. Should I have done that?
>
> could you please provide me with the test you have done together with
> the variables that you have set during for the computation? This would
> help me a lot.
>
> Cheers,
> Armando
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
Hi Sebastian,
On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
> No. Should I have done that?
could you please provide me with the test you have done together with
the variables that you have set during for the computation? This would
help me a lot.
Cheers,
Armando
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
It should be 24 per worker then. We cannot replicate your problem. Can
you give us exactly your setup and the command you run, and a log from one
worker?
On Thursday, February 13, 2014, Sebastian Schelter <ss...@apache.org> wrote:
> I'm using 24 workers to process a dataset of 50M vertices. Where can I see
> the number of partitions assigned to each worker?
>
> Best,
> Sebastian
>
> On 02/12/2014 04:34 PM, Claudio Martella wrote:
>
> If you set giraph.maxInMemoryPartitions accordingly to a number larger to
> that max(), and you're not setting stickyPartitions, then my question is
> whether there are at least that number of partitions in the worker. Can you
> validate whether there are enough partitions in the worker. In other terms,
> assuming you're using hash partitioner, how many workers are you using?
>
>
> On Wed, Feb 12, 2014 at 2:59 PM, Sebastian Schelter <ss...@apache.org>
> wrote:
>
> No. Should I have done that?
>
>
> On 02/12/2014 02:57 PM, Claudio Martella wrote:
>
> did you also set stickyPartitions to some numbers?
>
>
> On Wed, Feb 12, 2014 at 1:00 PM, Sebastian Schelter <ss...@apache.org>
> wrote:
>
> Updating documentation is never a bad idea :)
>
>
> I reran my test with giraph.maxPartitionsInMemory >
> max(giraph.numComputeThreads,giraph.numInputThreads,giraph.
> numOutputThreads)
> and still got the same behavior. I'll wait for the updated patch.
>
> Get well Armando!
>
>
> On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>
>
> btw: I as also thinking to update the documentation page on the Giraph
> website to better explain the sticky partition logic. What do you
> think?
>
> Cheers,
> Armando
>
>
> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>
> Indeed, yesterday I was fixing a couple of things and I think I missed
>
> a
> case that I have to exclude. Sorry for this, I have fever at the
> momento
> so it could be that yesterday I was under the effect of the fever :D
>
> I checked that the tests were passing but I think a missed something.
>
> I'll come back to you very soon
>
> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>
> the problem is that you're running with more threads than in-memory
>
> partitions. increase the number of partitions in memory to be at least
> the
> number of threads. i have no time right now to check the latest code,
> but
> you should not set the number of stickypartitions by hand.
>
>
> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org>
> wrote:
>
> I ran a first test with the new DiskBackedPartitionStore and it
> didn't
>
> work for me unfortunately. The job never leaves the input phase
> (superstep
> -1). I sshd onto one of the workers and it seems to wait forever on
> DiskBackedPartitionStore.getOrCreatePartition:
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:226)
> - waiting to lock <0x00000000aeb757c8> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>
> Here are the custom arguments for my run, let me know if I should do
> another run with a different config.
>
> giraph.oneToAllMsgSending=true
> giraph.isStaticGraph=true
> giraph.numComputeThreads=15
> giraph.numInputThreads=15
> giraph.numOutputThreads=15
> giraph.maxNumberOfSupersteps=30
> giraph.useOutOfCoreGraph=true
> giraph.stickyPartitions=5
>
> I also ran the job without using oneToAllMsgSending and saw the same
> behavior.
>
> Best,
> Sebastian
>
>
>
> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>
> please give it a test. i've been working on this with armando. i'll
>
> give a
> review, but we have been testing it for while. we'd really
> appreciate
> if
> somebody else could run some additional tests as well. thanks!
>
>
> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <
> ssc@apache.org>
> wrote:
>
> I'll test the patch from GIRAPH-825 this week.
>
>
>
> On 0
>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
I'm using 24 workers to process a dataset of 50M vertices. Where can I
see the number of partitions assigned to each worker?
Best,
Sebastian
On 02/12/2014 04:34 PM, Claudio Martella wrote:
> If you set giraph.maxInMemoryPartitions accordingly to a number larger to
> that max(), and you're not setting stickyPartitions, then my question is
> whether there are at least that number of partitions in the worker. Can you
> validate whether there are enough partitions in the worker. In other terms,
> assuming you're using hash partitioner, how many workers are you using?
>
>
> On Wed, Feb 12, 2014 at 2:59 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> No. Should I have done that?
>>
>>
>> On 02/12/2014 02:57 PM, Claudio Martella wrote:
>>
>>> did you also set stickyPartitions to some numbers?
>>>
>>>
>>> On Wed, Feb 12, 2014 at 1:00 PM, Sebastian Schelter <ss...@apache.org>
>>> wrote:
>>>
>>> Updating documentation is never a bad idea :)
>>>>
>>>> I reran my test with giraph.maxPartitionsInMemory >
>>>> max(giraph.numComputeThreads,giraph.numInputThreads,giraph.
>>>> numOutputThreads)
>>>> and still got the same behavior. I'll wait for the updated patch.
>>>>
>>>> Get well Armando!
>>>>
>>>>
>>>> On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>>>>
>>>>
>>>>> btw: I as also thinking to update the documentation page on the Giraph
>>>>> website to better explain the sticky partition logic. What do you
>>>>> think?
>>>>>
>>>>> Cheers,
>>>>> Armando
>>>>>
>>>>>
>>>>> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>>>>>
>>>>> Indeed, yesterday I was fixing a couple of things and I think I missed
>>>>>> a
>>>>>> case that I have to exclude. Sorry for this, I have fever at the
>>>>>> momento
>>>>>> so it could be that yesterday I was under the effect of the fever :D
>>>>>>
>>>>>> I checked that the tests were passing but I think a missed something.
>>>>>>
>>>>>> I'll come back to you very soon
>>>>>>
>>>>>> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>>>>>>
>>>>>> the problem is that you're running with more threads than in-memory
>>>>>>> partitions. increase the number of partitions in memory to be at least
>>>>>>> the
>>>>>>> number of threads. i have no time right now to check the latest code,
>>>>>>> but
>>>>>>> you should not set the number of stickypartitions by hand.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I ran a first test with the new DiskBackedPartitionStore and it
>>>>>>> didn't
>>>>>>>
>>>>>>>> work for me unfortunately. The job never leaves the input phase
>>>>>>>> (superstep
>>>>>>>> -1). I sshd onto one of the workers and it seems to wait forever on
>>>>>>>> DiskBackedPartitionStore.getOrCreatePartition:
>>>>>>>>
>>>>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>>>>>>>> getOrCreatePartition(DiskBackedPartitionStore.java:226)
>>>>>>>> - waiting to lock <0x00000000aeb757c8> (a
>>>>>>>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>>>>>>>>
>>>>>>>> Here are the custom arguments for my run, let me know if I should do
>>>>>>>> another run with a different config.
>>>>>>>>
>>>>>>>> giraph.oneToAllMsgSending=true
>>>>>>>> giraph.isStaticGraph=true
>>>>>>>> giraph.numComputeThreads=15
>>>>>>>> giraph.numInputThreads=15
>>>>>>>> giraph.numOutputThreads=15
>>>>>>>> giraph.maxNumberOfSupersteps=30
>>>>>>>> giraph.useOutOfCoreGraph=true
>>>>>>>> giraph.stickyPartitions=5
>>>>>>>>
>>>>>>>> I also ran the job without using oneToAllMsgSending and saw the same
>>>>>>>> behavior.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>>>>>>>>
>>>>>>>> please give it a test. i've been working on this with armando. i'll
>>>>>>>>
>>>>>>>>> give a
>>>>>>>>> review, but we have been testing it for while. we'd really
>>>>>>>>> appreciate
>>>>>>>>> if
>>>>>>>>> somebody else could run some additional tests as well. thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <
>>>>>>>>> ssc@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Given how big the diff here are:
>>>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>>>>>>> I am wondering whether it is realistic
>>>>>>>>>>> to have them in 1.1.0.
>>>>>>>>>>>
>>>>>>>>>>> Would appreciate folks chiming in.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Roman.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Claudio Martella
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
If you set giraph.maxInMemoryPartitions accordingly to a number larger to
that max(), and you're not setting stickyPartitions, then my question is
whether there are at least that number of partitions in the worker. Can you
validate whether there are enough partitions in the worker. In other terms,
assuming you're using hash partitioner, how many workers are you using?
On Wed, Feb 12, 2014 at 2:59 PM, Sebastian Schelter <ss...@apache.org> wrote:
> No. Should I have done that?
>
>
> On 02/12/2014 02:57 PM, Claudio Martella wrote:
>
>> did you also set stickyPartitions to some numbers?
>>
>>
>> On Wed, Feb 12, 2014 at 1:00 PM, Sebastian Schelter <ss...@apache.org>
>> wrote:
>>
>> Updating documentation is never a bad idea :)
>>>
>>> I reran my test with giraph.maxPartitionsInMemory >
>>> max(giraph.numComputeThreads,giraph.numInputThreads,giraph.
>>> numOutputThreads)
>>> and still got the same behavior. I'll wait for the updated patch.
>>>
>>> Get well Armando!
>>>
>>>
>>> On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>>>
>>>
>>>> btw: I as also thinking to update the documentation page on the Giraph
>>>> website to better explain the sticky partition logic. What do you
>>>> think?
>>>>
>>>> Cheers,
>>>> Armando
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>>>>
>>>> Indeed, yesterday I was fixing a couple of things and I think I missed
>>>>> a
>>>>> case that I have to exclude. Sorry for this, I have fever at the
>>>>> momento
>>>>> so it could be that yesterday I was under the effect of the fever :D
>>>>>
>>>>> I checked that the tests were passing but I think a missed something.
>>>>>
>>>>> I'll come back to you very soon
>>>>>
>>>>> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>>>>>
>>>>> the problem is that you're running with more threads than in-memory
>>>>>> partitions. increase the number of partitions in memory to be at least
>>>>>> the
>>>>>> number of threads. i have no time right now to check the latest code,
>>>>>> but
>>>>>> you should not set the number of stickypartitions by hand.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> I ran a first test with the new DiskBackedPartitionStore and it
>>>>>> didn't
>>>>>>
>>>>>>> work for me unfortunately. The job never leaves the input phase
>>>>>>> (superstep
>>>>>>> -1). I sshd onto one of the workers and it seems to wait forever on
>>>>>>> DiskBackedPartitionStore.getOrCreatePartition:
>>>>>>>
>>>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>>>>>>> getOrCreatePartition(DiskBackedPartitionStore.java:226)
>>>>>>> - waiting to lock <0x00000000aeb757c8> (a
>>>>>>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>>>>>>>
>>>>>>> Here are the custom arguments for my run, let me know if I should do
>>>>>>> another run with a different config.
>>>>>>>
>>>>>>> giraph.oneToAllMsgSending=true
>>>>>>> giraph.isStaticGraph=true
>>>>>>> giraph.numComputeThreads=15
>>>>>>> giraph.numInputThreads=15
>>>>>>> giraph.numOutputThreads=15
>>>>>>> giraph.maxNumberOfSupersteps=30
>>>>>>> giraph.useOutOfCoreGraph=true
>>>>>>> giraph.stickyPartitions=5
>>>>>>>
>>>>>>> I also ran the job without using oneToAllMsgSending and saw the same
>>>>>>> behavior.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>>>>>>>
>>>>>>> please give it a test. i've been working on this with armando. i'll
>>>>>>>
>>>>>>>> give a
>>>>>>>> review, but we have been testing it for while. we'd really
>>>>>>>> appreciate
>>>>>>>> if
>>>>>>>> somebody else could run some additional tests as well. thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <
>>>>>>>> ssc@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Given how big the diff here are:
>>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>>>>>> I am wondering whether it is realistic
>>>>>>>>>> to have them in 1.1.0.
>>>>>>>>>>
>>>>>>>>>> Would appreciate folks chiming in.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Roman.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Claudio Martella
>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
No. Should I have done that?
On 02/12/2014 02:57 PM, Claudio Martella wrote:
> did you also set stickyPartitions to some numbers?
>
>
> On Wed, Feb 12, 2014 at 1:00 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> Updating documentation is never a bad idea :)
>>
>> I reran my test with giraph.maxPartitionsInMemory >
>> max(giraph.numComputeThreads,giraph.numInputThreads,giraph.numOutputThreads)
>> and still got the same behavior. I'll wait for the updated patch.
>>
>> Get well Armando!
>>
>>
>> On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>>
>>>
>>> btw: I as also thinking to update the documentation page on the Giraph
>>> website to better explain the sticky partition logic. What do you
>>> think?
>>>
>>> Cheers,
>>> Armando
>>>
>>>
>>> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>>>
>>>> Indeed, yesterday I was fixing a couple of things and I think I missed a
>>>> case that I have to exclude. Sorry for this, I have fever at the momento
>>>> so it could be that yesterday I was under the effect of the fever :D
>>>>
>>>> I checked that the tests were passing but I think a missed something.
>>>>
>>>> I'll come back to you very soon
>>>>
>>>> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>>>>
>>>>> the problem is that you're running with more threads than in-memory
>>>>> partitions. increase the number of partitions in memory to be at least
>>>>> the
>>>>> number of threads. i have no time right now to check the latest code,
>>>>> but
>>>>> you should not set the number of stickypartitions by hand.
>>>>>
>>>>>
>>>>> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org>
>>>>> wrote:
>>>>>
>>>>> I ran a first test with the new DiskBackedPartitionStore and it didn't
>>>>>> work for me unfortunately. The job never leaves the input phase
>>>>>> (superstep
>>>>>> -1). I sshd onto one of the workers and it seems to wait forever on
>>>>>> DiskBackedPartitionStore.getOrCreatePartition:
>>>>>>
>>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>>>>>> getOrCreatePartition(DiskBackedPartitionStore.java:226)
>>>>>> - waiting to lock <0x00000000aeb757c8> (a
>>>>>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>>>>>>
>>>>>> Here are the custom arguments for my run, let me know if I should do
>>>>>> another run with a different config.
>>>>>>
>>>>>> giraph.oneToAllMsgSending=true
>>>>>> giraph.isStaticGraph=true
>>>>>> giraph.numComputeThreads=15
>>>>>> giraph.numInputThreads=15
>>>>>> giraph.numOutputThreads=15
>>>>>> giraph.maxNumberOfSupersteps=30
>>>>>> giraph.useOutOfCoreGraph=true
>>>>>> giraph.stickyPartitions=5
>>>>>>
>>>>>> I also ran the job without using oneToAllMsgSending and saw the same
>>>>>> behavior.
>>>>>>
>>>>>> Best,
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>>>>>>
>>>>>> please give it a test. i've been working on this with armando. i'll
>>>>>>> give a
>>>>>>> review, but we have been testing it for while. we'd really appreciate
>>>>>>> if
>>>>>>> somebody else could run some additional tests as well. thanks!
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Given how big the diff here are:
>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>>>>> I am wondering whether it is realistic
>>>>>>>>> to have them in 1.1.0.
>>>>>>>>>
>>>>>>>>> Would appreciate folks chiming in.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Roman.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>>
>>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
did you also set stickyPartitions to some numbers?
On Wed, Feb 12, 2014 at 1:00 PM, Sebastian Schelter <ss...@apache.org> wrote:
> Updating documentation is never a bad idea :)
>
> I reran my test with giraph.maxPartitionsInMemory >
> max(giraph.numComputeThreads,giraph.numInputThreads,giraph.numOutputThreads)
> and still got the same behavior. I'll wait for the updated patch.
>
> Get well Armando!
>
>
> On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>
>>
>> btw: I as also thinking to update the documentation page on the Giraph
>> website to better explain the sticky partition logic. What do you
>> think?
>>
>> Cheers,
>> Armando
>>
>>
>> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>>
>>> Indeed, yesterday I was fixing a couple of things and I think I missed a
>>> case that I have to exclude. Sorry for this, I have fever at the momento
>>> so it could be that yesterday I was under the effect of the fever :D
>>>
>>> I checked that the tests were passing but I think a missed something.
>>>
>>> I'll come back to you very soon
>>>
>>> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>>>
>>>> the problem is that you're running with more threads than in-memory
>>>> partitions. increase the number of partitions in memory to be at least
>>>> the
>>>> number of threads. i have no time right now to check the latest code,
>>>> but
>>>> you should not set the number of stickypartitions by hand.
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org>
>>>> wrote:
>>>>
>>>> I ran a first test with the new DiskBackedPartitionStore and it didn't
>>>>> work for me unfortunately. The job never leaves the input phase
>>>>> (superstep
>>>>> -1). I sshd onto one of the workers and it seems to wait forever on
>>>>> DiskBackedPartitionStore.getOrCreatePartition:
>>>>>
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>>>>> getOrCreatePartition(DiskBackedPartitionStore.java:226)
>>>>> - waiting to lock <0x00000000aeb757c8> (a
>>>>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>>>>>
>>>>> Here are the custom arguments for my run, let me know if I should do
>>>>> another run with a different config.
>>>>>
>>>>> giraph.oneToAllMsgSending=true
>>>>> giraph.isStaticGraph=true
>>>>> giraph.numComputeThreads=15
>>>>> giraph.numInputThreads=15
>>>>> giraph.numOutputThreads=15
>>>>> giraph.maxNumberOfSupersteps=30
>>>>> giraph.useOutOfCoreGraph=true
>>>>> giraph.stickyPartitions=5
>>>>>
>>>>> I also ran the job without using oneToAllMsgSending and saw the same
>>>>> behavior.
>>>>>
>>>>> Best,
>>>>> Sebastian
>>>>>
>>>>>
>>>>>
>>>>> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>>>>>
>>>>> please give it a test. i've been working on this with armando. i'll
>>>>>> give a
>>>>>> review, but we have been testing it for while. we'd really appreciate
>>>>>> if
>>>>>> somebody else could run some additional tests as well. thanks!
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>>>
>>>>>>>> Given how big the diff here are:
>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>>>> I am wondering whether it is realistic
>>>>>>>> to have them in 1.1.0.
>>>>>>>>
>>>>>>>> Would appreciate folks chiming in.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Roman.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>>
>>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
Updating documentation is never a bad idea :)
I reran my test with giraph.maxPartitionsInMemory >
max(giraph.numComputeThreads,giraph.numInputThreads,giraph.numOutputThreads)
and still got the same behavior. I'll wait for the updated patch.
Get well Armando!
On 02/12/2014 12:53 PM, Armando Miraglia wrote:
>
> btw: I as also thinking to update the documentation page on the Giraph
> website to better explain the sticky partition logic. What do you
> think?
>
> Cheers,
> Armando
>
>
> On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
>> Indeed, yesterday I was fixing a couple of things and I think I missed a
>> case that I have to exclude. Sorry for this, I have fever at the momento
>> so it could be that yesterday I was under the effect of the fever :D
>>
>> I checked that the tests were passing but I think a missed something.
>>
>> I'll come back to you very soon
>>
>> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
>>> the problem is that you're running with more threads than in-memory
>>> partitions. increase the number of partitions in memory to be at least the
>>> number of threads. i have no time right now to check the latest code, but
>>> you should not set the number of stickypartitions by hand.
>>>
>>>
>>> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org> wrote:
>>>
>>>> I ran a first test with the new DiskBackedPartitionStore and it didn't
>>>> work for me unfortunately. The job never leaves the input phase (superstep
>>>> -1). I sshd onto one of the workers and it seems to wait forever on
>>>> DiskBackedPartitionStore.getOrCreatePartition:
>>>>
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at org.apache.giraph.partition.DiskBackedPartitionStore.
>>>> getOrCreatePartition(DiskBackedPartitionStore.java:226)
>>>> - waiting to lock <0x00000000aeb757c8> (a
>>>> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>>>>
>>>> Here are the custom arguments for my run, let me know if I should do
>>>> another run with a different config.
>>>>
>>>> giraph.oneToAllMsgSending=true
>>>> giraph.isStaticGraph=true
>>>> giraph.numComputeThreads=15
>>>> giraph.numInputThreads=15
>>>> giraph.numOutputThreads=15
>>>> giraph.maxNumberOfSupersteps=30
>>>> giraph.useOutOfCoreGraph=true
>>>> giraph.stickyPartitions=5
>>>>
>>>> I also ran the job without using oneToAllMsgSending and saw the same
>>>> behavior.
>>>>
>>>> Best,
>>>> Sebastian
>>>>
>>>>
>>>>
>>>> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>>>>
>>>>> please give it a test. i've been working on this with armando. i'll give a
>>>>> review, but we have been testing it for while. we'd really appreciate if
>>>>> somebody else could run some additional tests as well. thanks!
>>>>>
>>>>>
>>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>>>>> wrote:
>>>>>
>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>>
>>>>>>
>>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>>
>>>>>> Hi!
>>>>>>>
>>>>>>> Given how big the diff here are:
>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>>> I am wondering whether it is realistic
>>>>>>> to have them in 1.1.0.
>>>>>>>
>>>>>>> Would appreciate folks chiming in.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Roman.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
btw: I as also thinking to update the documentation page on the Giraph
website to better explain the sticky partition logic. What do you
think?
Cheers,
Armando
On Wed, Feb 12, 2014 at 12:50:25PM +0100, Armando Miraglia wrote:
> Indeed, yesterday I was fixing a couple of things and I think I missed a
> case that I have to exclude. Sorry for this, I have fever at the momento
> so it could be that yesterday I was under the effect of the fever :D
>
> I checked that the tests were passing but I think a missed something.
>
> I'll come back to you very soon
>
> On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
> > the problem is that you're running with more threads than in-memory
> > partitions. increase the number of partitions in memory to be at least the
> > number of threads. i have no time right now to check the latest code, but
> > you should not set the number of stickypartitions by hand.
> >
> >
> > On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org> wrote:
> >
> > > I ran a first test with the new DiskBackedPartitionStore and it didn't
> > > work for me unfortunately. The job never leaves the input phase (superstep
> > > -1). I sshd onto one of the workers and it seems to wait forever on
> > > DiskBackedPartitionStore.getOrCreatePartition:
> > >
> > > java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.giraph.partition.DiskBackedPartitionStore.
> > > getOrCreatePartition(DiskBackedPartitionStore.java:226)
> > > - waiting to lock <0x00000000aeb757c8> (a
> > > org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> > >
> > > Here are the custom arguments for my run, let me know if I should do
> > > another run with a different config.
> > >
> > > giraph.oneToAllMsgSending=true
> > > giraph.isStaticGraph=true
> > > giraph.numComputeThreads=15
> > > giraph.numInputThreads=15
> > > giraph.numOutputThreads=15
> > > giraph.maxNumberOfSupersteps=30
> > > giraph.useOutOfCoreGraph=true
> > > giraph.stickyPartitions=5
> > >
> > > I also ran the job without using oneToAllMsgSending and saw the same
> > > behavior.
> > >
> > > Best,
> > > Sebastian
> > >
> > >
> > >
> > > On 02/12/2014 12:44 AM, Claudio Martella wrote:
> > >
> > >> please give it a test. i've been working on this with armando. i'll give a
> > >> review, but we have been testing it for while. we'd really appreciate if
> > >> somebody else could run some additional tests as well. thanks!
> > >>
> > >>
> > >> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
> > >> wrote:
> > >>
> > >> I'll test the patch from GIRAPH-825 this week.
> > >>>
> > >>>
> > >>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
> > >>>
> > >>> Hi!
> > >>>>
> > >>>> Given how big the diff here are:
> > >>>> https://issues.apache.org/jira/browse/GIRAPH-825
> > >>>> https://issues.apache.org/jira/browse/GIRAPH-840
> > >>>> I am wondering whether it is realistic
> > >>>> to have them in 1.1.0.
> > >>>>
> > >>>> Would appreciate folks chiming in.
> > >>>>
> > >>>> Thanks,
> > >>>> Roman.
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
> > --
> > Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Armando Miraglia <a....@student.vu.nl>.
Indeed, yesterday I was fixing a couple of things and I think I missed a
case that I have to exclude. Sorry for this, I have fever at the momento
so it could be that yesterday I was under the effect of the fever :D
I checked that the tests were passing but I think a missed something.
I'll come back to you very soon
On Wed, Feb 12, 2014 at 10:26:12AM +0100, Claudio Martella wrote:
> the problem is that you're running with more threads than in-memory
> partitions. increase the number of partitions in memory to be at least the
> number of threads. i have no time right now to check the latest code, but
> you should not set the number of stickypartitions by hand.
>
>
> On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org> wrote:
>
> > I ran a first test with the new DiskBackedPartitionStore and it didn't
> > work for me unfortunately. The job never leaves the input phase (superstep
> > -1). I sshd onto one of the workers and it seems to wait forever on
> > DiskBackedPartitionStore.getOrCreatePartition:
> >
> > java.lang.Thread.State: BLOCKED (on object monitor)
> > at org.apache.giraph.partition.DiskBackedPartitionStore.
> > getOrCreatePartition(DiskBackedPartitionStore.java:226)
> > - waiting to lock <0x00000000aeb757c8> (a
> > org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
> >
> > Here are the custom arguments for my run, let me know if I should do
> > another run with a different config.
> >
> > giraph.oneToAllMsgSending=true
> > giraph.isStaticGraph=true
> > giraph.numComputeThreads=15
> > giraph.numInputThreads=15
> > giraph.numOutputThreads=15
> > giraph.maxNumberOfSupersteps=30
> > giraph.useOutOfCoreGraph=true
> > giraph.stickyPartitions=5
> >
> > I also ran the job without using oneToAllMsgSending and saw the same
> > behavior.
> >
> > Best,
> > Sebastian
> >
> >
> >
> > On 02/12/2014 12:44 AM, Claudio Martella wrote:
> >
> >> please give it a test. i've been working on this with armando. i'll give a
> >> review, but we have been testing it for while. we'd really appreciate if
> >> somebody else could run some additional tests as well. thanks!
> >>
> >>
> >> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
> >> wrote:
> >>
> >> I'll test the patch from GIRAPH-825 this week.
> >>>
> >>>
> >>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
> >>>
> >>> Hi!
> >>>>
> >>>> Given how big the diff here are:
> >>>> https://issues.apache.org/jira/browse/GIRAPH-825
> >>>> https://issues.apache.org/jira/browse/GIRAPH-840
> >>>> I am wondering whether it is realistic
> >>>> to have them in 1.1.0.
> >>>>
> >>>> Would appreciate folks chiming in.
> >>>>
> >>>> Thanks,
> >>>> Roman.
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
> --
> Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
the problem is that you're running with more threads than in-memory
partitions. increase the number of partitions in memory to be at least the
number of threads. i have no time right now to check the latest code, but
you should not set the number of stickypartitions by hand.
On Wed, Feb 12, 2014 at 10:03 AM, Sebastian Schelter <ss...@apache.org> wrote:
> I ran a first test with the new DiskBackedPartitionStore and it didn't
> work for me unfortunately. The job never leaves the input phase (superstep
> -1). I sshd onto one of the workers and it seems to wait forever on
> DiskBackedPartitionStore.getOrCreatePartition:
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:226)
> - waiting to lock <0x00000000aeb757c8> (a
> org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
>
> Here are the custom arguments for my run, let me know if I should do
> another run with a different config.
>
> giraph.oneToAllMsgSending=true
> giraph.isStaticGraph=true
> giraph.numComputeThreads=15
> giraph.numInputThreads=15
> giraph.numOutputThreads=15
> giraph.maxNumberOfSupersteps=30
> giraph.useOutOfCoreGraph=true
> giraph.stickyPartitions=5
>
> I also ran the job without using oneToAllMsgSending and saw the same
> behavior.
>
> Best,
> Sebastian
>
>
>
> On 02/12/2014 12:44 AM, Claudio Martella wrote:
>
>> please give it a test. i've been working on this with armando. i'll give a
>> review, but we have been testing it for while. we'd really appreciate if
>> somebody else could run some additional tests as well. thanks!
>>
>>
>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>> wrote:
>>
>> I'll test the patch from GIRAPH-825 this week.
>>>
>>>
>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>
>>> Hi!
>>>>
>>>> Given how big the diff here are:
>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>> I am wondering whether it is realistic
>>>> to have them in 1.1.0.
>>>>
>>>> Would appreciate folks chiming in.
>>>>
>>>> Thanks,
>>>> Roman.
>>>>
>>>>
>>>>
>>>
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
I ran a first test with the new DiskBackedPartitionStore and it didn't
work for me unfortunately. The job never leaves the input phase
(superstep -1). I sshd onto one of the workers and it seems to wait
forever on DiskBackedPartitionStore.getOrCreatePartition:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:226)
- waiting to lock <0x00000000aeb757c8> (a
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
Here are the custom arguments for my run, let me know if I should do
another run with a different config.
giraph.oneToAllMsgSending=true
giraph.isStaticGraph=true
giraph.numComputeThreads=15
giraph.numInputThreads=15
giraph.numOutputThreads=15
giraph.maxNumberOfSupersteps=30
giraph.useOutOfCoreGraph=true
giraph.stickyPartitions=5
I also ran the job without using oneToAllMsgSending and saw the same
behavior.
Best,
Sebastian
On 02/12/2014 12:44 AM, Claudio Martella wrote:
> please give it a test. i've been working on this with armando. i'll give a
> review, but we have been testing it for while. we'd really appreciate if
> somebody else could run some additional tests as well. thanks!
>
>
> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> I'll test the patch from GIRAPH-825 this week.
>>
>>
>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>
>>> Hi!
>>>
>>> Given how big the diff here are:
>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>> I am wondering whether it is realistic
>>> to have them in 1.1.0.
>>>
>>> Would appreciate folks chiming in.
>>>
>>> Thanks,
>>> Roman.
>>>
>>>
>>
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
Well, as RM I am only tracking things tagged with 1.1.0 -- feel free to tag
whatever you think needs to be in the release.
I'll be making a final^n call for patches over the weekend. Next
week, I'd really like to start testing and only allow bug fixes into
the release branches.
Thanks,
Roman.
On Tue, Feb 11, 2014 at 3:54 PM, Avery Ching <ac...@apache.org> wrote:
> We haven't been using it at FB yet, it's a great step in that direction
> though.
>
>
> On 2/11/14, 3:52 PM, Claudio Martella wrote:
>>
>> One additional issue that might go in could be an add-on to the
>> documentation about the python interface. I remember nitay mentioned not
>> to
>> do a big fuzz about it because it was not heavily tested. Maybe the
>> facebook people can say something about it? If reliable, this would be one
>> of the major points of this release, IMO.
>>
>>
>> On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik
>> <ro...@shaposhnik.org>wrote:
>>
>>> Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
>>> manageable now:
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
>>>
>>> I am really counting on being able to cut release branches
>>> over the weekend.
>>>
>>>
>>> Thanks,
>>> Roman.
>>>
>>> On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
>>> <cl...@gmail.com> wrote:
>>>>
>>>> please give it a test. i've been working on this with armando. i'll give
>>>
>>> a
>>>>
>>>> review, but we have been testing it for while. we'd really appreciate if
>>>> somebody else could run some additional tests as well. thanks!
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>>>
>>> wrote:
>>>>>
>>>>> I'll test the patch from GIRAPH-825 this week.
>>>>>
>>>>>
>>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Given how big the diff here are:
>>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>>> I am wondering whether it is realistic
>>>>>> to have them in 1.1.0.
>>>>>>
>>>>>> Would appreciate folks chiming in.
>>>>>>
>>>>>> Thanks,
>>>>>> Roman.
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>
>>
>>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Avery Ching <ac...@apache.org>.
We haven't been using it at FB yet, it's a great step in that direction
though.
On 2/11/14, 3:52 PM, Claudio Martella wrote:
> One additional issue that might go in could be an add-on to the
> documentation about the python interface. I remember nitay mentioned not to
> do a big fuzz about it because it was not heavily tested. Maybe the
> facebook people can say something about it? If reliable, this would be one
> of the major points of this release, IMO.
>
>
> On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
>
>> Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
>> manageable now:
>>
>> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
>>
>> I am really counting on being able to cut release branches
>> over the weekend.
>>
>>
>> Thanks,
>> Roman.
>>
>> On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
>> <cl...@gmail.com> wrote:
>>> please give it a test. i've been working on this with armando. i'll give
>> a
>>> review, but we have been testing it for while. we'd really appreciate if
>>> somebody else could run some additional tests as well. thanks!
>>>
>>>
>>> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
>> wrote:
>>>> I'll test the patch from GIRAPH-825 this week.
>>>>
>>>>
>>>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> Given how big the diff here are:
>>>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>>>> I am wondering whether it is realistic
>>>>> to have them in 1.1.0.
>>>>>
>>>>> Would appreciate folks chiming in.
>>>>>
>>>>> Thanks,
>>>>> Roman.
>>>>>
>>>>>
>>>
>>> --
>>> Claudio Martella
>
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
This issue is blocked for the moment, so we ll sort it out separately.
On Tuesday, March 4, 2014, Mirko Kämpf <mi...@cloudera.com> wrote:
> Hi Claudio,
>
> is there some material I can merge into the docu, or is there a JIRA to
> track this addition to the documentation?
>
> Cheers,
> Mirko
>
>
>
> On Wed, Feb 12, 2014 at 12:52 AM, Claudio Martella <
> claudio.martella@gmail.com<javascript:_e(%7B%7D,'cvml','claudio.martella@gmail.com');>
> > wrote:
>
>> One additional issue that might go in could be an add-on to the
>> documentation about the python interface. I remember nitay mentioned not
>> to
>> do a big fuzz about it because it was not heavily tested. Maybe the
>> facebook people can say something about it? If reliable, this would be one
>> of the major points of this release, IMO.
>>
>>
>> On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik <roman@shaposhnik.org<javascript:_e(%7B%7D,'cvml','roman@shaposhnik.org');>
>> >wrote:
>>
>> > Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
>> > manageable now:
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
>> >
>> > I am really counting on being able to cut release branches
>> > over the weekend.
>> >
>> >
>> > Thanks,
>> > Roman.
>> >
>> > On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
>> > <claudio.martella@gmail.com<javascript:_e(%7B%7D,'cvml','claudio.martella@gmail.com');>>
>> wrote:
>> > > please give it a test. i've been working on this with armando. i'll
>> give
>> > a
>> > > review, but we have been testing it for while. we'd really appreciate
>> if
>> > > somebody else could run some additional tests as well. thanks!
>> > >
>> > >
>> > > On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ssc@apache.org<javascript:_e(%7B%7D,'cvml','ssc@apache.org');>
>> >
>> > wrote:
>> > >
>> > >> I'll test the patch from GIRAPH-825 this week.
>> > >>
>> > >>
>> > >> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>> > >>
>> > >>> Hi!
>> > >>>
>> > >>> Given how big the diff here are:
>> > >>> https://issues.apache.org/jira/browse/GIRAPH-825
>> > >>> https://issues.apache.org/jira/browse/GIRAPH-840
>> > >>> I am wondering whether it is realistic
>> > >>> to have them in 1.1.0.
>> > >>>
>> > >>> Would appreciate folks chiming in.
>> > >>>
>> > >>> Thanks,
>> > >>> Roman.
>> > >>>
>> > >>>
>> > >>
>> > >
>> > >
>> > > --
>> > > Claudio Martella
>> >
>>
>>
>>
>> --
>> Claudio Martella
>>
>
>
>
> --
> --
> Mirko Kämpf
>
> *Trainer* @ Cloudera
>
> tel: +49 *176 20 63 51 99*
> skype: *kamir1604*
> mirko@cloudera.com <javascript:_e(%7B%7D,'cvml','mirko@cloudera.com');>
>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
This issue is blocked for the moment, so we ll sort it out separately.
On Tuesday, March 4, 2014, Mirko Kämpf <mi...@cloudera.com> wrote:
> Hi Claudio,
>
> is there some material I can merge into the docu, or is there a JIRA to
> track this addition to the documentation?
>
> Cheers,
> Mirko
>
>
>
> On Wed, Feb 12, 2014 at 12:52 AM, Claudio Martella <
> claudio.martella@gmail.com<javascript:_e(%7B%7D,'cvml','claudio.martella@gmail.com');>
> > wrote:
>
>> One additional issue that might go in could be an add-on to the
>> documentation about the python interface. I remember nitay mentioned not
>> to
>> do a big fuzz about it because it was not heavily tested. Maybe the
>> facebook people can say something about it? If reliable, this would be one
>> of the major points of this release, IMO.
>>
>>
>> On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik <roman@shaposhnik.org<javascript:_e(%7B%7D,'cvml','roman@shaposhnik.org');>
>> >wrote:
>>
>> > Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
>> > manageable now:
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
>> >
>> > I am really counting on being able to cut release branches
>> > over the weekend.
>> >
>> >
>> > Thanks,
>> > Roman.
>> >
>> > On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
>> > <claudio.martella@gmail.com<javascript:_e(%7B%7D,'cvml','claudio.martella@gmail.com');>>
>> wrote:
>> > > please give it a test. i've been working on this with armando. i'll
>> give
>> > a
>> > > review, but we have been testing it for while. we'd really appreciate
>> if
>> > > somebody else could run some additional tests as well. thanks!
>> > >
>> > >
>> > > On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ssc@apache.org<javascript:_e(%7B%7D,'cvml','ssc@apache.org');>
>> >
>> > wrote:
>> > >
>> > >> I'll test the patch from GIRAPH-825 this week.
>> > >>
>> > >>
>> > >> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>> > >>
>> > >>> Hi!
>> > >>>
>> > >>> Given how big the diff here are:
>> > >>> https://issues.apache.org/jira/browse/GIRAPH-825
>> > >>> https://issues.apache.org/jira/browse/GIRAPH-840
>> > >>> I am wondering whether it is realistic
>> > >>> to have them in 1.1.0.
>> > >>>
>> > >>> Would appreciate folks chiming in.
>> > >>>
>> > >>> Thanks,
>> > >>> Roman.
>> > >>>
>> > >>>
>> > >>
>> > >
>> > >
>> > > --
>> > > Claudio Martella
>> >
>>
>>
>>
>> --
>> Claudio Martella
>>
>
>
>
> --
> --
> Mirko Kämpf
>
> *Trainer* @ Cloudera
>
> tel: +49 *176 20 63 51 99*
> skype: *kamir1604*
> mirko@cloudera.com <javascript:_e(%7B%7D,'cvml','mirko@cloudera.com');>
>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Mirko Kämpf <mi...@cloudera.com>.
Hi Claudio,
is there some material I can merge into the docu, or is there a JIRA to
track this addition to the documentation?
Cheers,
Mirko
On Wed, Feb 12, 2014 at 12:52 AM, Claudio Martella <
claudio.martella@gmail.com> wrote:
> One additional issue that might go in could be an add-on to the
> documentation about the python interface. I remember nitay mentioned not to
> do a big fuzz about it because it was not heavily tested. Maybe the
> facebook people can say something about it? If reliable, this would be one
> of the major points of this release, IMO.
>
>
> On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik <roman@shaposhnik.org
> >wrote:
>
> > Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
> > manageable now:
> >
> >
> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
> >
> > I am really counting on being able to cut release branches
> > over the weekend.
> >
> >
> > Thanks,
> > Roman.
> >
> > On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
> > <cl...@gmail.com> wrote:
> > > please give it a test. i've been working on this with armando. i'll
> give
> > a
> > > review, but we have been testing it for while. we'd really appreciate
> if
> > > somebody else could run some additional tests as well. thanks!
> > >
> > >
> > > On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
> > wrote:
> > >
> > >> I'll test the patch from GIRAPH-825 this week.
> > >>
> > >>
> > >> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
> > >>
> > >>> Hi!
> > >>>
> > >>> Given how big the diff here are:
> > >>> https://issues.apache.org/jira/browse/GIRAPH-825
> > >>> https://issues.apache.org/jira/browse/GIRAPH-840
> > >>> I am wondering whether it is realistic
> > >>> to have them in 1.1.0.
> > >>>
> > >>> Would appreciate folks chiming in.
> > >>>
> > >>> Thanks,
> > >>> Roman.
> > >>>
> > >>>
> > >>
> > >
> > >
> > > --
> > > Claudio Martella
> >
>
>
>
> --
> Claudio Martella
>
--
--
Mirko Kämpf
*Trainer* @ Cloudera
tel: +49 *176 20 63 51 99*
skype: *kamir1604*
mirko@cloudera.com
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
One additional issue that might go in could be an add-on to the
documentation about the python interface. I remember nitay mentioned not to
do a big fuzz about it because it was not heavily tested. Maybe the
facebook people can say something about it? If reliable, this would be one
of the major points of this release, IMO.
On Wed, Feb 12, 2014 at 12:47 AM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
> manageable now:
>
> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
>
> I am really counting on being able to cut release branches
> over the weekend.
>
>
> Thanks,
> Roman.
>
> On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
> <cl...@gmail.com> wrote:
> > please give it a test. i've been working on this with armando. i'll give
> a
> > review, but we have been testing it for while. we'd really appreciate if
> > somebody else could run some additional tests as well. thanks!
> >
> >
> > On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org>
> wrote:
> >
> >> I'll test the patch from GIRAPH-825 this week.
> >>
> >>
> >> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
> >>
> >>> Hi!
> >>>
> >>> Given how big the diff here are:
> >>> https://issues.apache.org/jira/browse/GIRAPH-825
> >>> https://issues.apache.org/jira/browse/GIRAPH-840
> >>> I am wondering whether it is realistic
> >>> to have them in 1.1.0.
> >>>
> >>> Would appreciate folks chiming in.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>>
> >>
> >
> >
> > --
> > Claudio Martella
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
Great! The list of JIRAs scheduled to go into 1.1.0 looks pretty
manageable now:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%221.1.0%22%20AND%20project%20%3D%20GIRAPH%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20status%20DESC%2C%20summary%20ASC%2C%20priority%20DESC
I am really counting on being able to cut release branches
over the weekend.
Thanks,
Roman.
On Tue, Feb 11, 2014 at 3:44 PM, Claudio Martella
<cl...@gmail.com> wrote:
> please give it a test. i've been working on this with armando. i'll give a
> review, but we have been testing it for while. we'd really appreciate if
> somebody else could run some additional tests as well. thanks!
>
>
> On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> I'll test the patch from GIRAPH-825 this week.
>>
>>
>> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>>
>>> Hi!
>>>
>>> Given how big the diff here are:
>>> https://issues.apache.org/jira/browse/GIRAPH-825
>>> https://issues.apache.org/jira/browse/GIRAPH-840
>>> I am wondering whether it is realistic
>>> to have them in 1.1.0.
>>>
>>> Would appreciate folks chiming in.
>>>
>>> Thanks,
>>> Roman.
>>>
>>>
>>
>
>
> --
> Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Claudio Martella <cl...@gmail.com>.
please give it a test. i've been working on this with armando. i'll give a
review, but we have been testing it for while. we'd really appreciate if
somebody else could run some additional tests as well. thanks!
On Wed, Feb 12, 2014 at 12:39 AM, Sebastian Schelter <ss...@apache.org> wrote:
> I'll test the patch from GIRAPH-825 this week.
>
>
> On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
>
>> Hi!
>>
>> Given how big the diff here are:
>> https://issues.apache.org/jira/browse/GIRAPH-825
>> https://issues.apache.org/jira/browse/GIRAPH-840
>> I am wondering whether it is realistic
>> to have them in 1.1.0.
>>
>> Would appreciate folks chiming in.
>>
>> Thanks,
>> Roman.
>>
>>
>
--
Claudio Martella
Re: GIRAPH-825 and GIRAPH-840
Posted by Sebastian Schelter <ss...@apache.org>.
I'll test the patch from GIRAPH-825 this week.
On 02/12/2014 12:10 AM, Roman Shaposhnik wrote:
> Hi!
>
> Given how big the diff here are:
> https://issues.apache.org/jira/browse/GIRAPH-825
> https://issues.apache.org/jira/browse/GIRAPH-840
> I am wondering whether it is realistic
> to have them in 1.1.0.
>
> Would appreciate folks chiming in.
>
> Thanks,
> Roman.
>
Re: GIRAPH-825 and GIRAPH-840
Posted by Avery Ching <ac...@apache.org>.
GIRAPH-840 will be committed very soon I think. It's ready to go.
On 2/11/14, 3:10 PM, Roman Shaposhnik wrote:
> Hi!
>
> Given how big the diff here are:
> https://issues.apache.org/jira/browse/GIRAPH-825
> https://issues.apache.org/jira/browse/GIRAPH-840
> I am wondering whether it is realistic
> to have them in 1.1.0.
>
> Would appreciate folks chiming in.
>
> Thanks,
> Roman.