Re: [malhar-users] Caching in Datatorrent

[Thomas Weise <th...@gmail.com>]

An operator can use any technology to implement the cache. Here is an
example how to structure a cache from the Malhar library:

https://github.com/apache/incubator-apex-malhar/blob/v3.0.0/library/src/main/java/com/datatorrent/lib/db/cache/AbstractDBLookupCacheBackedOperator.java


Please note that the resulting caching is local to each operator instance
(every partition has its own cache). The idea is that access to shared
state is minimized for more efficient processing.


On Mon, Aug 3, 2015 at 8:21 AM, Amol Kekre <am...@datatorrent.com> wrote:

>
> Chiranjeevi,
> We do have guava (
> https://code.google.com/p/guava-libraries/wiki/CachesExplained) available
> as a Malhar operator
>
>
> https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
>
> Do take a look.
>
> Thks,
> Amol
>
>
> On Mon, Aug 3, 2015 at 1:51 AM, ravi <ch...@gmail.com> wrote:
>
>> Hi Team,
>>
>> Do we have caching available in Datatorrent? If yes, Please let me know
>> how to implement.
>>
>> -many thanks
>> Chiranjeevi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Malhar" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to malhar-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to malhar-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/malhar-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Malhar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to malhar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to malhar-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/malhar-users.
> For more options, visit https://groups.google.com/d/optout.
>

Re: [malhar-users] Caching in Datatorrent

[Siyuan Hua <si...@datatorrent.com>]

 The cache manager
https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
is a cache used only for DTFile. DTFile is a TFileReader(Writer) with cache
which provides much faster IO compared to native hadoop TFile but keeps the
format



On Tue, Sep 1, 2015 at 2:58 PM, Thomas Weise <th...@gmail.com> wrote:

> An operator can use any technology to implement the cache. Here is an
> example how to structure a cache from the Malhar library:
>
>
> https://github.com/apache/incubator-apex-malhar/blob/v3.0.0/library/src/main/java/com/datatorrent/lib/db/cache/AbstractDBLookupCacheBackedOperator.java
>
>
> Please note that the resulting caching is local to each operator instance
> (every partition has its own cache). The idea is that access to shared
> state is minimized for more efficient processing.
>
>
> On Mon, Aug 3, 2015 at 8:21 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
> >
> > Chiranjeevi,
> > We do have guava (
> > https://code.google.com/p/guava-libraries/wiki/CachesExplained)
> available
> > as a Malhar operator
> >
> >
> >
> https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
> >
> > Do take a look.
> >
> > Thks,
> > Amol
> >
> >
> > On Mon, Aug 3, 2015 at 1:51 AM, ravi <ch...@gmail.com> wrote:
> >
> >> Hi Team,
> >>
> >> Do we have caching available in Datatorrent? If yes, Please let me know
> >> how to implement.
> >>
> >> -many thanks
> >> Chiranjeevi
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Malhar" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to malhar-users+unsubscribe@googlegroups.com.
> >> To post to this group, send email to malhar-users@googlegroups.com.
> >> Visit this group at http://groups.google.com/group/malhar-users.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Malhar" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to malhar-users+unsubscribe@googlegroups.com.
> > To post to this group, send email to malhar-users@googlegroups.com.
> > Visit this group at http://groups.google.com/group/malhar-users.
> > For more options, visit https://groups.google.com/d/optout.
> >
>