You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by lee fei <ch...@gmail.com> on 2012/12/01 05:41:11 UTC
Re: Can Aggregator store a queue or a map for global communication?
hi Thomas,
I want to use an aggregator to store an HashMap, and in every superstep,
different vertexes may put or remove an element<key, value> to the HashMap,
and I am wondering at the sync phase,if the master will automatic sync the
HashMap?
Just like this:
superstep n: HashMap: 1,2,3,4,5,6
task1 modified the aggregator:
HashMap(1): 3,4,5,6(task1 remove 1,2)
task2 modified the aggregator:
HashMap(2): 1,2,3,4,5,6,7,8(task2 add 7,8)
....
and after the sync phase also in next superstep:
the HashMap become: 3,4,5,6,7,8
Did I got it? the namenode will automatic do this?
On Thu, Nov 29, 2012 at 8:52 PM, lee fei <ch...@gmail.com> wrote:
> Thanks for you quick reply:)
>
> Best wishes for you:D
>
>
> On Thu, Nov 29, 2012 at 8:01 PM, Thomas Jungblut <
> thomas.jungblut@gmail.com> wrote:
>
>> Yes, if they are Writable and are part of your vertex message or value
>> type.
>> You may want to have a look at MapWritable or other serializable
>> collections in the org.apache.hadoop.io package.
>>
>> 2012/11/29 lee fei <ch...@gmail.com>
>>
>> > Hi,
>> >
>> > I am a newbie to hama,Can Aggregator in graph package store an
>> Arraylist
>> > or a HashMap for some special usages?
>> >
>> > Thank you for you attention~
>> >
>>
>
>
Re: Can Aggregator store a queue or a map for global communication?
Posted by lee fei <ch...@gmail.com>.
Yes, I find the MapWritable in org.apache.hadoop.io, it's very useful.
Thank you very much!
On Sat, Dec 1, 2012 at 4:38 PM, Thomas Jungblut
<th...@gmail.com>wrote:
> Like I told you, the aggregator works through the message type of your
> vertex. This must be Writable, so if you find a HashMap that is Writable
> (maybe you have to write your own) and this is your message type in your
> algorithm. Then this will work automatically if you implement your
> reduction operations correctly in the aggregator.
>
>
>
> 2012/12/1 lee fei <ch...@gmail.com>
>
> > hi Thomas,
> > I want to use an aggregator to store an HashMap, and in every
> superstep,
> > different vertexes may put or remove an element<key, value> to the
> HashMap,
> > and I am wondering at the sync phase,if the master will automatic sync
> the
> > HashMap?
> >
> > Just like this:
> >
> > superstep n: HashMap: 1,2,3,4,5,6
> >
> > task1 modified the aggregator:
> > HashMap(1): 3,4,5,6(task1 remove 1,2)
> >
> > task2 modified the aggregator:
> > HashMap(2): 1,2,3,4,5,6,7,8(task2 add 7,8)
> > ....
> > and after the sync phase also in next superstep:
> > the HashMap become: 3,4,5,6,7,8
> >
> > Did I got it? the namenode will automatic do this?
> >
> >
> > On Thu, Nov 29, 2012 at 8:52 PM, lee fei <ch...@gmail.com> wrote:
> >
> > > Thanks for you quick reply:)
> > >
> > > Best wishes for you:D
> > >
> > >
> > > On Thu, Nov 29, 2012 at 8:01 PM, Thomas Jungblut <
> > > thomas.jungblut@gmail.com> wrote:
> > >
> > >> Yes, if they are Writable and are part of your vertex message or value
> > >> type.
> > >> You may want to have a look at MapWritable or other serializable
> > >> collections in the org.apache.hadoop.io package.
> > >>
> > >> 2012/11/29 lee fei <ch...@gmail.com>
> > >>
> > >> > Hi,
> > >> >
> > >> > I am a newbie to hama,Can Aggregator in graph package store an
> > >> Arraylist
> > >> > or a HashMap for some special usages?
> > >> >
> > >> > Thank you for you attention~
> > >> >
> > >>
> > >
> > >
> >
>
Re: Can Aggregator store a queue or a map for global communication?
Posted by Thomas Jungblut <th...@gmail.com>.
Like I told you, the aggregator works through the message type of your
vertex. This must be Writable, so if you find a HashMap that is Writable
(maybe you have to write your own) and this is your message type in your
algorithm. Then this will work automatically if you implement your
reduction operations correctly in the aggregator.
2012/12/1 lee fei <ch...@gmail.com>
> hi Thomas,
> I want to use an aggregator to store an HashMap, and in every superstep,
> different vertexes may put or remove an element<key, value> to the HashMap,
> and I am wondering at the sync phase,if the master will automatic sync the
> HashMap?
>
> Just like this:
>
> superstep n: HashMap: 1,2,3,4,5,6
>
> task1 modified the aggregator:
> HashMap(1): 3,4,5,6(task1 remove 1,2)
>
> task2 modified the aggregator:
> HashMap(2): 1,2,3,4,5,6,7,8(task2 add 7,8)
> ....
> and after the sync phase also in next superstep:
> the HashMap become: 3,4,5,6,7,8
>
> Did I got it? the namenode will automatic do this?
>
>
> On Thu, Nov 29, 2012 at 8:52 PM, lee fei <ch...@gmail.com> wrote:
>
> > Thanks for you quick reply:)
> >
> > Best wishes for you:D
> >
> >
> > On Thu, Nov 29, 2012 at 8:01 PM, Thomas Jungblut <
> > thomas.jungblut@gmail.com> wrote:
> >
> >> Yes, if they are Writable and are part of your vertex message or value
> >> type.
> >> You may want to have a look at MapWritable or other serializable
> >> collections in the org.apache.hadoop.io package.
> >>
> >> 2012/11/29 lee fei <ch...@gmail.com>
> >>
> >> > Hi,
> >> >
> >> > I am a newbie to hama,Can Aggregator in graph package store an
> >> Arraylist
> >> > or a HashMap for some special usages?
> >> >
> >> > Thank you for you attention~
> >> >
> >>
> >
> >
>