You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by David Wimsey <dw...@bandwidth.com> on 2017/11/21 16:19:57 UTC
WAL size control
Hello all,
I’m having an issue understanding how to control/limit the size of the WAL as well as why I’m consuming so much WAL space.
I’m currently migrating a project from redis to ignite and as part of the first phase, its pretty much lift and shift without optimizing for ignite. In this case, I’m passing in a handful of large objects to a cache, most of these are Map<String, String> types, but one is a more complex structure. All of these objects combined consume a few GB of RAM but the maps themselves may contain tens of thousands of entries, though at this stage I don’t care if ignite is aware of these objects, when I retrieve them from the cache I simply pull out the full object (see code below). I realize this is not ideal, and will be refactored to work more efficiently in the next phase.
private IgniteCache<String, Serializable> objectCache;
public LookupDataCacheProvider(final Ignite igniteDataGrid)
{
this.igniteDataGrid = igniteDataGrid;
metadataCache = igniteDataGrid.getOrCreateCache(LookupDataCacheKey.METADATA.getKey());
objectCache = igniteDataGrid.getOrCreateCache(OBJECT_CACHE_NAME).withKeepBinary();
}
public void setRateCenters(final RateCenters rateCenters)
{
setJavaObject(LookupDataCacheKey.RATE_CENTERS, rateCenters);
}
private <T extends Serializable> void setJavaObject(final LookupDataCacheKey lookupDataCacheKey, final T value) {
objectCache.put(lookupDataCacheKey.getKey(), value);
}
private <T extends Serializable> T getJavaObject(final LookupDataCacheKey lookupDataCacheKey, final Class<T> tClass) {
return tClass.cast(objectCache.get(lookupDataCacheKey.getKey()));
}
My problem is that when the largest of these maps get written to the cache, it fills up the mount we have dedicated to persistent storage, looking at the directory structure the majority of the space is in WAL archives.
We have a 100G mount for persistence, which we assumed would be plenty for something that normally fits into a few GB of ram.
So, can anyone point me at something I can use to calculate the size needed for the WAL based on ignite settings or what settings I really care about? Additionally, can anyone give me a general reason for why the WAL may get so massive for such a small set of data (max 4G data in ram, 100gb in WAL before it runs out of space). Is this just a case of using ignite so wrong that it doesn’t work? If I were to make the Map<String, String> a cache by itself rather than a object inside a cache, as a general rule would that be more likely or less likely to increase WAL usage.
What am I doing wrong that is consuming so much WAL?
Re: WAL size control
Posted by Denis Magda <dm...@apache.org>.
Justin, check this section, you can tun WAL or disable WAL archiving in
general:
https://apacheignite.readme.io/docs/write-ahead-log#section-tuning-wal-archive
-
Denis
On Mon, Jan 14, 2019 at 7:06 PM Justin Ji <bi...@foxmail.com> wrote:
> I have more than three hundred WAL segments in our folder
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2000/3.png>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: WAL size control
Posted by Justin Ji <bi...@foxmail.com>.
I have more than three hundred WAL segments in our folder
<http://apache-ignite-users.70518.x6.nabble.com/file/t2000/3.png>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: WAL size control
Posted by Justin Ji <bi...@foxmail.com>.
Hi Igniters -
I have the problem too, but I only have one node in the server.
I want to know how to delete the unused WAL segments safety.
Looking forward to your reply.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: WAL size control
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi David!
WAL <https://apacheignite.readme.io/docs/write-ahead-log> is a ring buffer
on the filesystem. Each file in it is called a "segment". By default each
segment occupies 64 MB on disk (*walSegmentSize* property), and there are
10 such files in the buffer (*walSegments* property). So, WAL for each node
may grow up to ~640 MB.
But there is also a WAL archive. When a segment is full, it is copied to
the archive. And the number of files in the archive depends on the *walHistSize
*property, which is 20 checkpoints by default. It specifies the number of
checkpoints, that the history should keep.
Checkpoint <https://apacheignite.readme.io/docs/persistence-checkpointing>
is a procedure of writing data to the persistent disk space. It is
triggered when it didn't happen for a long time, or a lot of updates
happened since the last checkpoint.
So, you can tune these properties, and it may lead to disk usage decrease.
But don't set any of these to very low values. You can find here how you
can tune durable memory:
https://apacheignite.readme.io/docs/durable-memory-tuning
Note, that changing these values may lead to that nodes won't be able to
read WAL segments, that are already stored on disk.
If WAL directory grows up to 100GB, it means, that you either run many
nodes on the same machine, or old unused WAL segments are still stored
there. Check, that it doesn't contain WAL for nodes, that are not in the
cluster.
Denis
вт, 21 нояб. 2017 г. в 19:20, David Wimsey <dw...@bandwidth.com>:
> Hello all,
>
> I’m having an issue understanding how to control/limit the size of the WAL
> as well as why I’m consuming so much WAL space.
>
> I’m currently migrating a project from redis to ignite and as part of the
> first phase, its pretty much lift and shift without optimizing for ignite.
> In this case, I’m passing in a handful of large objects to a cache, most of
> these are Map<String, String> types, but one is a more complex structure.
> All of these objects combined consume a few GB of RAM but the maps
> themselves may contain tens of thousands of entries, though at this stage I
> don’t care if ignite is aware of these objects, when I retrieve them from
> the cache I simply pull out the full object (see code below). I realize
> this is not ideal, and will be refactored to work more efficiently in the
> next phase.
>
> private IgniteCache<String, Serializable> objectCache;
>
> public LookupDataCacheProvider(final Ignite igniteDataGrid)
> {
> this.igniteDataGrid = igniteDataGrid;
> metadataCache = igniteDataGrid.getOrCreateCache(LookupDataCacheKey.METADATA.getKey());
> objectCache = igniteDataGrid.getOrCreateCache(OBJECT_CACHE_NAME).withKeepBinary();
> }
>
> public void setRateCenters(final RateCenters rateCenters)
> {
>
> setJavaObject(LookupDataCacheKey.RATE_CENTERS, rateCenters);
> }
>
> private <T extends Serializable> void setJavaObject(final LookupDataCacheKey
> lookupDataCacheKey, final T value) {
>
> objectCache.put(lookupDataCacheKey.getKey(), value);
> }
>
> private <T extends Serializable> T getJavaObject(final LookupDataCacheKey lookupDataCacheKey, final Class<T> tClass) {
> return tClass.cast(objectCache.get(lookupDataCacheKey.getKey()));
> }
>
>
> My problem is that when the largest of these maps get written to the
> cache, it fills up the mount we have dedicated to persistent storage,
> looking at the directory structure the majority of the space is in WAL
> archives.
>
> We have a 100G mount for persistence, which we assumed would be plenty for
> something that normally fits into a few GB of ram.
>
> So, can anyone point me at something I can use to calculate the size
> needed for the WAL based on ignite settings or what settings I really care
> about? Additionally, can anyone give me a general reason for why the WAL
> may get so massive for such a small set of data (max 4G data in ram, 100gb
> in WAL before it runs out of space). Is this just a case of using ignite
> so wrong that it doesn’t work? If I were to make the Map<String, String> a
> cache by itself rather than a object inside a cache, as a general rule
> would that be more likely or less likely to increase WAL usage.
>
> What am I doing wrong that is consuming so much WAL?
>