Spooled Data Structures

[Timothy Farkas <ti...@datatorrent.com>]

Hi all,

There is currently a need for Spooled Lists and Heaps. I'd like to be able
to leverage Chandni's large state management or other key value stores
without changing the list or heap implementation. In order to do this I'd
like a Store interface which can be passed to a Spooled Data structure and
used transparently. I was thinking the following for an interface

public interface AddressableStore<VALUE>
{
  public VALUE get(long address);
  public void set(long address, VALUE value);
  public void free(long address);
}

A data structure would then take this and use it to store it's data

  public SpooledArray(AddressableStore<VALUE> addressableStore)
  {
    this.addressableStore = Preconditions.checkNotNull(addressableStore);
  }

  public void add(VALUE value)
  {
    size++;
    addressableStore.set(size, value);
  }

Thoughts?

Thanks,
Tim

Re: Spooled Data Structures

[Chandni Singh <ch...@datatorrent.com>]

Hi Tim,

+1
I think this is a very good idea. The only recommendation I have is to work
on this after ManagedState is reviewed and merged to Malhar.
IMO an interface needs to be backed by multiple implementations (at least
one).  The AddressableState ( Store is too generic, just nitpicking) needs
an implementation which can be an extension of ManagedStateImpl.

The example you provided is of a SpooledList rather than a SpooledArray
(again nitpicking :-) ).

Chandni


On Tue, Dec 8, 2015 at 12:00 PM, Timothy Farkas <ti...@datatorrent.com> wrote:

> There's a bug :)
>
> should be
>
>   public void add(VALUE value)
>   {
>     addressableStore.set(size, value);
>     size++;
>   }
>
> On Tue, Dec 8, 2015 at 11:57 AM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
> > Hi all,
> >
> > There is currently a need for Spooled Lists and Heaps. I'd like to be
> able
> > to leverage Chandni's large state management or other key value stores
> > without changing the list or heap implementation. In order to do this I'd
> > like a Store interface which can be passed to a Spooled Data structure
> and
> > used transparently. I was thinking the following for an interface
> >
> > public interface AddressableStore<VALUE>
> > {
> >   public VALUE get(long address);
> >   public void set(long address, VALUE value);
> >   public void free(long address);
> > }
> >
> > A data structure would then take this and use it to store it's data
> >
> >   public SpooledArray(AddressableStore<VALUE> addressableStore)
> >   {
> >     this.addressableStore = Preconditions.checkNotNull(addressableStore);
> >   }
> >
> >   public void add(VALUE value)
> >   {
> >     size++;
> >     addressableStore.set(size, value);
> >   }
> >
> > Thoughts?
> >
> > Thanks,
> > Tim
> >
>

Re: Spooled Data Structures

[Timothy Farkas <ti...@datatorrent.com>]

There's a bug :)

should be

  public void add(VALUE value)
  {
    addressableStore.set(size, value);
    size++;
  }

On Tue, Dec 8, 2015 at 11:57 AM, Timothy Farkas <ti...@datatorrent.com> wrote:

> Hi all,
>
> There is currently a need for Spooled Lists and Heaps. I'd like to be able
> to leverage Chandni's large state management or other key value stores
> without changing the list or heap implementation. In order to do this I'd
> like a Store interface which can be passed to a Spooled Data structure and
> used transparently. I was thinking the following for an interface
>
> public interface AddressableStore<VALUE>
> {
>   public VALUE get(long address);
>   public void set(long address, VALUE value);
>   public void free(long address);
> }
>
> A data structure would then take this and use it to store it's data
>
>   public SpooledArray(AddressableStore<VALUE> addressableStore)
>   {
>     this.addressableStore = Preconditions.checkNotNull(addressableStore);
>   }
>
>   public void add(VALUE value)
>   {
>     size++;
>     addressableStore.set(size, value);
>   }
>
> Thoughts?
>
> Thanks,
> Tim
>