Re: Window Commits

[Vlad Rozov <v....@datatorrent.com>]

Atri,

The buffer server operates on data blocks, not on time windows. Tuples 
with the same time window are not guaranteed to belong to the same 
buffer server data block.

There are few enhancements left open after I moved persistence of data 
blocks to a different thread:

1. When persisting a block keep reference to in-memory data if number of 
in-memory blocks does not exceed the maximum allowed
2. Introduce option to spool a block to disk when X or more blocks have 
in-memory reference
3. Improve block eviction selection

Thank you,

Vlad

On 8/26/15 04:17, Atri Sharma wrote:
> Vlad,
>
> Thinking more about this I think that the right way to look at this is to
> use the patch discussed below that you wrote and make it flexible for any
> time window (rather than half as your patch does).
>
> Thoughts?
> On 25 Aug 2015 06:33, "Vlad Rozov" <v....@datatorrent.com> wrote:
>
>> To add to Chetan comment, I am working on an enhancement that offloads
>> bufferserver spooling to disk to a separate thread. This thread will kick
>> in when only half (should it be configurable?) of memory blocks are left
>> available. This should further avoid possibility of I/O spikes.
>>
>> Vlad
>>
>> On 8/24/15 14:16, Chetan Narsude wrote:
>>
>>> The bufferserver writes pages to disk *only when* it runs out of memory to
>>> hold them.
>>>
>>> Can you elaborate where you see I/O spikes?
>>>
>>> --
>>> Chetan
>>>
>>> On Mon, Aug 24, 2015 at 12:39 PM, Atri Sharma <at...@gmail.com>
>>> wrote:
>>>
>>> Folks,
>>>> I was wondering if it makes sense to have a functionality in which
>>>> bufferserver writes out data pages to disk in batches defined by
>>>> timeslice/application window.
>>>>
>>>> This will allow flexible workloads and reduce I/O spikes (I understand
>>>> that
>>>> we have non-blocking I/O but it still would incur disk head costs).
>>>>
>>>> Thoughts?
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Atri
>>>> *l'apprenant*
>>>>
>>>>