receiving messages that I didn't send

[Matthew Cornell <ma...@matthewcornell.org>]

Hi Folks. I am refactoring my compute()  to use a set of ids as its
message type, and in my tests it is receiving a message that it
absolutely did not send. I've debugged it and am at a loss.
Interestingly, I encountered this once before and solved it by
creating a copy of a Writeable instead of re-using it, but I haven't
been able to solve it this time. In general, does this anomalous
behavior indicate a Giraph/Hadoop gotcha'? It's really confounding!
Thank very much -- matt

-- 
Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34
Dickinson Street, Amherst MA 01002 | matthewcornell.org

Re: receiving messages that I didn't send

[Matthew Cornell <ma...@matthewcornell.org>]

Thanks for replying, Pavan. I figured out that my message Writable (an
ArrayListWritable) needed to call clear() in readFields() before
calling super():

    @Override
    public void readFields(DataInput in) throws IOException {
        clear();
        super.readFields(in);
    }

This was an 'of course' moment when I realized it was, like other
Writables, being reused. But what I don't understand is why doesn't
ArrayListWritable#readFields() call clear? Isn't this a nasty bug? ...
Oh wait - sure enough:

ArrayListWritable object is not cleared in readFields()
https://issues.apache.org/jira/browse/GIRAPH-740

Thanks again,

matt


On Tue, Sep 23, 2014 at 11:46 AM, Pavan Kumar A <pa...@outlook.com> wrote:
> Can you give more context?
> What are the types of messages, patch of your compute method, etc.
> You will not receive messages that are not sent, but one thing that can
> happen is
> -- message can have multiple parameters.
> suppose message objects can have 2 parameters
> m - a,b
> say in m's write(out) you do not handle the case of b = null
> m1 sets b
> m2 has b=null
> then because of incorrect code for m's write() m2 can show b = m1.b
> that is because message objects will be re-used when receiving. This is a
> Giraph gotcha, because of
> object reuse in most iterators.
>
> Thanks
>
>> From: matt@matthewcornell.org
>> Date: Tue, 23 Sep 2014 10:10:48 -0400
>> Subject: receiving messages that I didn't send
>> To: user@giraph.apache.org
>
>>
>> Hi Folks. I am refactoring my compute() to use a set of ids as its
>> message type, and in my tests it is receiving a message that it
>> absolutely did not send. I've debugged it and am at a loss.
>> Interestingly, I encountered this once before and solved it by
>> creating a copy of a Writeable instead of re-using it, but I haven't
>> been able to solve it this time. In general, does this anomalous
>> behavior indicate a Giraph/Hadoop gotcha'? It's really confounding!
>> Thank very much -- matt
>>
>> --
>> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34
>> Dickinson Street, Amherst MA 01002 | matthewcornell.org



-- 
Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34
Dickinson Street, Amherst MA 01002 | matthewcornell.org

RE: receiving messages that I didn't send

[Pavan Kumar A <pa...@outlook.com>]

Can you give more context?What are the types of messages, patch of your compute method, etc.You will not receive messages that are not sent, but one thing that can happen is-- message can have multiple parameters.suppose message objects can have 2 parametersm - a,bsay in m's write(out) you do not handle the case of b = nullm1 sets bm2 has b=nullthen because of incorrect code for m's write() m2 can show b = m1.bthat is because message objects will be re-used when receiving. This is a Giraph gotcha, because ofobject reuse in most iterators.
Thanks

> From: matt@matthewcornell.org
> Date: Tue, 23 Sep 2014 10:10:48 -0400
> Subject: receiving messages that I didn't send
> To: user@giraph.apache.org
> 
> Hi Folks. I am refactoring my compute()  to use a set of ids as its
> message type, and in my tests it is receiving a message that it
> absolutely did not send. I've debugged it and am at a loss.
> Interestingly, I encountered this once before and solved it by
> creating a copy of a Writeable instead of re-using it, but I haven't
> been able to solve it this time. In general, does this anomalous
> behavior indicate a Giraph/Hadoop gotcha'? It's really confounding!
> Thank very much -- matt
> 
> -- 
> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34
> Dickinson Street, Amherst MA 01002 | matthewcornell.org