You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2011/02/14 14:05:34 UTC
Re: communication
Oh, thanks for nice report!
At this time, it looks like impossible to send one more messages on to
same server in bsp() method. It's a bug. I'll fix it now.
Pls, update your TRUNK.
----
To developers,
See the BSPPeer.send() method. The peerName should be a
InetSocketAddress, not String. Otherwise, queue will be always null.
And after add msg to queue, we have to put it to "outgoingQueues"
again as below:
--- src/java/org/apache/hama/bsp/BSPPeer.java (revision 1069189)
+++ src/java/org/apache/hama/bsp/BSPPeer.java (working copy)
@@ -140,12 +140,12 @@
@Override
public void send(String peerName, BSPMessage msg) throws IOException {
LOG.debug("Send bytes (" + msg.getData().toString() + ") to " + peerName);
- ConcurrentLinkedQueue<BSPMessage> queue = outgoingQueues.get(peerName);
+ ConcurrentLinkedQueue<BSPMessage> queue =
outgoingQueues.get(getAddress(peerName));
if (queue == null) {
queue = new ConcurrentLinkedQueue<BSPMessage>();
- outgoingQueues.put(getAddress(peerName), queue);
}
queue.add(msg);
+ outgoingQueues.put(getAddress(peerName), queue);
}
On Mon, Feb 14, 2011 at 7:02 PM, Paweł Brach <br...@gmail.com> wrote:
> Hello,
>
> I'm attaching PiEstimator with some changes. I'm sending exactly two
> messages to "master" from each node.
>
> bspPeer.send(masterTask, estimate);
> LOG.info("Send message:" + System.currentTimeMillis());
> bspPeer.send(masterTask, estimate2);
> LOG.info("Send message:" + System.currentTimeMillis());
>
> After that I'm trying to receive all messages:
> LOG.info("Num msg = " + bspPeer.getNumCurrentMessages());
> while ((received = bspPeer.getCurrentMessage()) != null) {
> LOG.info("Receives messages:" + Bytes.toDouble(received.getData()));
> ....
> }
>
> In the log file I see:
> 2011-02-14 09:46:07,124 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator: Send message:1297673167123
> 2011-02-14 09:46:07,125 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator: Send message:1297673167125
> 2011-02-14 09:46:07,270 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator: Receives messages:3.1536
> 2011-02-14 09:46:07,270 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator: Receives messages:3.148
> 2011-02-14 09:46:07,271 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator: Receives messages:3.1392
> 2011-02-14 09:46:07,272 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 11/02/14 09:46:07 INFO
> algorithms.PiEstimator$MyEstimator:
> 2011-02-14 09:46:07,272 INFO org.apache.hama.bsp.TaskRunner:
> attempt_201102140938_0003_m_000000_0 Estimated value of PI is
> 3.1450000000000005
>
>
> On each node I see that two messages are sent (so master should receive 6
> messages - I have 3 nodes). In the log file I see only 3 records
> "Receives...", so only 3 messages were processed by master task. I don'
> understand why is that.
>
> What's more, I have tried to change line:
> while ((received = bspPeer.getCurrentMessage()) != null) {
>
> to
> if ((received = bspPeer.getCurrentMessage()) != null) {
>
> So there should be messages in the queue which are not processed. In my
> opinion bsp() should be called again and again... and in consequence falls
> in loop of calls. But it doesn't happen - bsp() method is run exactly once
> and framework is probably hanging after last line of the bsp() code. Could
> you explain this for me?
>
> Best regards,
> Pawel
>
>
>
> 2011/2/10 Edward J. Yoon <ed...@apache.org>
>>
>> It should be possible to send multiple messages to other nodes.
>>
>> Could you please show me your source code and debug level log?
>>
>
--
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon
Re: communication
Posted by "Edward J. Yoon" <ed...@apache.org>.
> So, the bsp() method is called once per whole job ?
> I thought that one superstep = one bsp() call.
Oh, .... yes, right. I'm going to fix some document and code error.
I was designed the bsp() function, which user can whole BSP program
ideally - https://issues.apache.org/jira/browse/HAMA-257 - but, I
confused with Map function of MapReduce or Google Pregel, too. :/
On Tue, Feb 15, 2011 at 4:39 PM, Paweł Brach <br...@gmail.com> wrote:
> So, the bsp() method is called once per whole job ?
> I thought that one superstep = one bsp() call.
> I'm trying to find an example when the bsp() method is called more than once
> on each node.
>
> Thanks.
>
> 2011/2/15 Edward J. Yoon <ed...@apache.org>
>
>> You can understand it as a barrier count. After all peers have entered
>> the barrier using sync() method, the system proceeds to the next
>> superstep.
>>
>> And, even if queues are empty, the bsp job won't stop until each
>> processes terminated. In SerializePrinting example case, the program
>> will be finished after the for loop is done and the count of
>> supersteps is same with a number of peers.
>>
>> Thanks.
>>
>>
>
--
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon
Re: communication
Posted by "Edward J. Yoon" <ed...@apache.org>.
> So, the bsp() method is called once per whole job ?
> I thought that one superstep = one bsp() call.
Oh, .... yes, right. I'm going to fix some document and code error.
I was designed the bsp() function, which user can whole BSP program
ideally - https://issues.apache.org/jira/browse/HAMA-257 - but, I
confused with Map function of MapReduce or Google Pregel, too. :/
On Tue, Feb 15, 2011 at 4:39 PM, Paweł Brach <br...@gmail.com> wrote:
> So, the bsp() method is called once per whole job ?
> I thought that one superstep = one bsp() call.
> I'm trying to find an example when the bsp() method is called more than once
> on each node.
>
> Thanks.
>
> 2011/2/15 Edward J. Yoon <ed...@apache.org>
>
>> You can understand it as a barrier count. After all peers have entered
>> the barrier using sync() method, the system proceeds to the next
>> superstep.
>>
>> And, even if queues are empty, the bsp job won't stop until each
>> processes terminated. In SerializePrinting example case, the program
>> will be finished after the for loop is done and the count of
>> supersteps is same with a number of peers.
>>
>> Thanks.
>>
>>
>
--
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon
Re: communication
Posted by Paweł Brach <br...@gmail.com>.
So, the bsp() method is called once per whole job ?
I thought that one superstep = one bsp() call.
I'm trying to find an example when the bsp() method is called more than once
on each node.
Thanks.
2011/2/15 Edward J. Yoon <ed...@apache.org>
> You can understand it as a barrier count. After all peers have entered
> the barrier using sync() method, the system proceeds to the next
> superstep.
>
> And, even if queues are empty, the bsp job won't stop until each
> processes terminated. In SerializePrinting example case, the program
> will be finished after the for loop is done and the count of
> supersteps is same with a number of peers.
>
> Thanks.
>
>
Re: communication
Posted by "Edward J. Yoon" <ed...@apache.org>.
You can understand it as a barrier count. After all peers have entered
the barrier using sync() method, the system proceeds to the next
superstep.
And, even if queues are empty, the bsp job won't stop until each
processes terminated. In SerializePrinting example case, the program
will be finished after the for loop is done and the count of
supersteps is same with a number of peers.
Thanks.
On Mon, Feb 14, 2011 at 11:29 PM, Paweł Brach <br...@gmail.com> wrote:
> Thanks for quick bugfix. Could you provide me an example code, where you
> control number of supersteps by sending messages to itself ?
> I'm really confused looking at SerializePrinting example provided with Hama.
> There is no send message and there are 3 supersteps.
> It looks like the number of supersteps in controlled by number of loop
> iterations in the bsp() method.
> What's more, in the javadoc there is info about sync method:
> "Sends all the messages in the outgoing message queues to the corresponding
> remote peers."
> So what is the reason to call sync() in the SerializePrinting example
> (outgoing queue is empty).
>
> Thanks,
> Pawel
>
>
>
> 2011/2/14 Edward J. Yoon <ed...@apache.org>
>
>> Oh, thanks for nice report!
>>
>> At this time, it looks like impossible to send one more messages on to
>> same server in bsp() method. It's a bug. I'll fix it now.
>>
>> Pls, update your TRUNK.
>>
>>
>
--
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon
Re: communication
Posted by Paweł Brach <br...@gmail.com>.
Thanks for quick bugfix. Could you provide me an example code, where you
control number of supersteps by sending messages to itself ?
I'm really confused looking at SerializePrinting example provided with Hama.
There is no send message and there are 3 supersteps.
It looks like the number of supersteps in controlled by number of loop
iterations in the bsp() method.
What's more, in the javadoc there is info about sync method:
"Sends all the messages in the outgoing message queues to the corresponding
remote peers."
So what is the reason to call sync() in the SerializePrinting example
(outgoing queue is empty).
Thanks,
Pawel
2011/2/14 Edward J. Yoon <ed...@apache.org>
> Oh, thanks for nice report!
>
> At this time, it looks like impossible to send one more messages on to
> same server in bsp() method. It's a bug. I'll fix it now.
>
> Pls, update your TRUNK.
>
>