Re: Native support for huge files and range requests

[Alan Carroll <so...@yahoo-inc.com.INVALID>]

I think it would be possible, it's something that been discussed before. You would also need to replace CacheVC which is used as the interface between the cache the virtual I/O system inside ATS. I haven't investigated that to any serious extent so it's hard to say the overall difficulty. You would have replace the entire cache subsystem, the current internals are not very compatible with this.

On Monday, January 30, 2017 6:52 AM, Ori Finkelman <or...@qwilt.com> wrote:

*Writing to cache*
We need to be able to write huge files and be able to receive them in
chunks (ranges) as they are received over separated requests. When writing
a chunk of a file we pre-allocate sequential space for the next chunks.
Small chunks are buffered and written to disk only when enough is
aggregated across separate requests. This also inherently solves the http
ranges problem as space is always reserved for ranges which were not yet
acquired.

*Reading from cache*
When reading an object we use read-ahead into memory to optimize on IOPS as
in most cases the next chunks of the objects will be requested within a few
seconds.

*Evacuation*
Evacuation should be done based on popularity (for example LRU), and works
for the whole object together, not per chunk.
The cyclone-buffer is not ideal for such a use case as the evacuation
policy is different and we can't pay for the the copy of large objects
every time the write cursor pass them.


Since objects are very large and writing out is done in very large chunks,
file system overhead is negligible and therefore an architecture relying on
Linux file system may have benefits, such as the ability to read objects
from FS rather than via HTTP for debug and analysis.
It is also possible to implement something which is based on the existing
cache architecture with the required adaptations.

We are trying to evaluate if implementing such a new cache storage mode is
possible ? is it reasonable ? were there previous thoughts in such
directions ?
I could find in the wiki reference for both large files and ranges
requests, but no conclusions.

In a fast glimpse, It does look like it would not be very hard to replace
the existing *CacheProcessor *and *CacheVConnection *with different ones
which will implement a different cache storage.

Thank you in advance for any feedback.

-- 

*Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 |
orif@qwilt.com

Re: Native support for huge files and range requests

[Ori Finkelman <or...@qwilt.com>]

Thanks Alan.

On Mon, Feb 6, 2017, 16:53 Alan Carroll <so...@yahoo-inc.com>
wrote:

> Let me know how it goes and feel free to reach out in you have any further
> questions.
>
-- 

*Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 |
orif@qwilt.com

Re: Native support for huge files and range requests

[Alan Carroll <so...@yahoo-inc.com.INVALID>]

Let me know how it goes and feel free to reach out in you have any further questions.

Re: Native support for huge files and range requests

[Ori Finkelman <or...@qwilt.com>]

Thank you Alan.
We will take a deeper look and see where it goes.

Best regards,
Ori

On Wed, Feb 1, 2017 at 4:35 PM, Alan Carroll <solidwallofcode@yahoo-inc.
com.invalid> wrote:

> I think it would be possible, it's something that been discussed before.
> You would also need to replace CacheVC which is used as the interface
> between the cache the virtual I/O system inside ATS. I haven't investigated
> that to any serious extent so it's hard to say the overall difficulty. You
> would have replace the entire cache subsystem, the current internals are
> not very compatible with this.
>
> On Monday, January 30, 2017 6:52 AM, Ori Finkelman <or...@qwilt.com> wrote:
>
> *Writing to cache*
> We need to be able to write huge files and be able to receive them in
> chunks (ranges) as they are received over separated requests. When writing
> a chunk of a file we pre-allocate sequential space for the next chunks.
> Small chunks are buffered and written to disk only when enough is
> aggregated across separate requests. This also inherently solves the http
> ranges problem as space is always reserved for ranges which were not yet
> acquired.
>
> *Reading from cache*
> When reading an object we use read-ahead into memory to optimize on IOPS as
> in most cases the next chunks of the objects will be requested within a few
> seconds.
>
> *Evacuation*
> Evacuation should be done based on popularity (for example LRU), and works
> for the whole object together, not per chunk.
> The cyclone-buffer is not ideal for such a use case as the evacuation
> policy is different and we can't pay for the the copy of large objects
> every time the write cursor pass them.
>
>
> Since objects are very large and writing out is done in very large chunks,
> file system overhead is negligible and therefore an architecture relying on
> Linux file system may have benefits, such as the ability to read objects
> from FS rather than via HTTP for debug and analysis.
> It is also possible to implement something which is based on the existing
> cache architecture with the required adaptations.
>
> We are trying to evaluate if implementing such a new cache storage mode is
> possible ? is it reasonable ? were there previous thoughts in such
> directions ?
> I could find in the wiki reference for both large files and ranges
> requests, but no conclusions.
>
> In a fast glimpse, It does look like it would not be very hard to replace
> the existing *CacheProcessor *and *CacheVConnection *with different ones
> which will implement a different cache storage.
>
> Thank you in advance for any feedback.
>
> --
>
> *Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 |
> orif@qwilt.com
>



-- 

*Ori Finkelman*Qwilt | Work: +972-72-2221647 <072-222-1647> | Mobile:
+972-52-3832189 <052-383-2189> | orif@qwilt.com