Re: eviction performance

[Scott Feldstein <sc...@gmail.com>]

Hi Stan,
Thanks for the follow up.  The part that I'm confused about here is when an
expired entry isn't accessed it would simply linger on disk forever.

In my prototype I want to be able to store one continuous hour of hot data
to be queryable at all times.  The volume of data currently sits at 20k
entries / second but I'd like to increase this substantially.
Unfortunately the cleanup issue getting in the way of this.

To load up the data I'm using the ignite streamer, this makes it more
difficult to create a new cache in order to clean up old data (not
impossible but awkward).  I'm using the streamer since in the docs it says
that is the most efficient way to load data into the system.  Is there a
better way to load up the data for my use case other than using a streamer?

In my scenario I would like to be able to have a light weight way of
cleaning up the data.  While the data may not be cleaned up immediately I'd
like to have it eventually cleaned up since it is taking up a lot of disk.
Is this possible without using a separate cache?  Or is using multiple
caches the only way you can think of to achieve this?

Do you think that a "truncate partition" feature is something that would be
useful / doable in Ignite?  Would it be worth a feature request?

thanks!

On Sun, Mar 25, 2018 at 1:59 PM Stanislav Lukyanov <st...@gmail.com>
wrote:

> Hi Scott,
>
> With eagetTTL=false, each time you access an entry its TTL is automatically
> checked - if it is expired, the entry will be removed. It means that
> instead
> of having a separate thread waking up and removing all of the expired
> entries at the same time (and taking a chunk of CPU and IO for a noticeable
> period of time), you'll be removing them over time. On average, individual
> accesses may become a bit more expensive (because some of them will trigger
> removal) but that will allow you to avoid latency spikes.
>
> About the truncate - I think there is no API to do that based on
> partitions.
> I guess you could try creating a new cache for each batch and destroying it
> after you don't need it anymore, but it's hard to say whether it will be
> more efficient.
>
> Stan
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>