Can I share datas for several map tasks?

[snowloong <sn...@gmail.com>]

Hi,
I want to share some data structures for the map tasks on a same node(not through files), I mean, if one map task has already initialized some data structures (e.g. an array or a list), can other map tasks share these memorys and directly access them, for I don't want to reinitialize these datas and I want to save some memory. Can hadoop help me do this?
thanks

2009-06-16 



snowloong 

Re: Can I share datas for several map tasks?

["Eason.Lee" <le...@gmail.com>]

Sorry, I think I have mistook your problem.I thought you want share data in
the different calls of the map() method in the same MapTask
Hope anyone else can help you [?]

2009/6/16 Hello World <sn...@gmail.com>

> Thanks for your reply. I tried as you said that init some data in setup()
> and read them from map(), it did work, but it still seems that each map
> task
> did its own setup() seperately and they are using different datas, so this
> does not match what I need. I want to share some common datas amongst
> different map tasks (these datas are read-only after initialization). So is
> there any other way?
>
>
> On Tue, Jun 16, 2009 at 9:28 AM, Eason.Lee <le...@gmail.com> wrote:
>
> > I think you can just define the data structures in your map classinit it
> in
> > setup(Context context) and use it in your map method
> > hope it is helpful!
> >
> > 2009/6/16 snowloong <sn...@gmail.com>
> >
> > > Hi,
> > > I want to share some data structures for the map tasks on a same
> node(not
> > > through files), I mean, if one map task has already initialized some
> data
> > > structures (e.g. an array or a list), can other map tasks share these
> > > memorys and directly access them, for I don't want to reinitialize
> these
> > > datas and I want to save some memory. Can hadoop help me do this?
> > > thanks
> > >
> > > 2009-06-16
> > >
> > >
> > >
> > > snowloong
> > >
> >
>

Re: Can I share datas for several map tasks?

[Hello World <sn...@gmail.com>]

Thanks for your reply. I tried as you said that init some data in setup()
and read them from map(), it did work, but it still seems that each map task
did its own setup() seperately and they are using different datas, so this
does not match what I need. I want to share some common datas amongst
different map tasks (these datas are read-only after initialization). So is
there any other way?


On Tue, Jun 16, 2009 at 9:28 AM, Eason.Lee <le...@gmail.com> wrote:

> I think you can just define the data structures in your map classinit it in
> setup(Context context) and use it in your map method
> hope it is helpful!
>
> 2009/6/16 snowloong <sn...@gmail.com>
>
> > Hi,
> > I want to share some data structures for the map tasks on a same node(not
> > through files), I mean, if one map task has already initialized some data
> > structures (e.g. an array or a list), can other map tasks share these
> > memorys and directly access them, for I don't want to reinitialize these
> > datas and I want to save some memory. Can hadoop help me do this?
> > thanks
> >
> > 2009-06-16
> >
> >
> >
> > snowloong
> >
>

Re: Can I share datas for several map tasks?

["Eason.Lee" <le...@gmail.com>]

I think you can just define the data structures in your map classinit it in
setup(Context context) and use it in your map method
hope it is helpful!

2009/6/16 snowloong <sn...@gmail.com>

> Hi,
> I want to share some data structures for the map tasks on a same node(not
> through files), I mean, if one map task has already initialized some data
> structures (e.g. an array or a list), can other map tasks share these
> memorys and directly access them, for I don't want to reinitialize these
> datas and I want to save some memory. Can hadoop help me do this?
> thanks
>
> 2009-06-16
>
>
>
> snowloong
>