number of way segments in wal

[KR Kumar <kr...@gmail.com>]

Hi Guys - For some weird reason, files are filling up in wal folder and
right now I have around 65,000 files and occupies almost 4.5TB disk.
Ideally it should not be more than 10 files right? Also i have disabled wal
archiving. Why is this happening? and what am i missing?

Following is my configuration regarding wal configuration

<property name="dataStorageConfiguration">

<bean class="org.apache.ignite.configuration.DataStorageConfiguration">

<property name="writeThrottlingEnabled" value="true"></property>

<property name="defaultDataRegionConfiguration">

<bean class="org.apache.ignite.configuration.DataRegionConfiguration">

<property name="initialSize" value="#{512L *  1024 * 1024}"></property>

<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />

<property name="persistenceEnabled" value="true" />

<property name="checkpointPageBufferSize" value="#{1024 * 1024 * 1024}" />

</bean>

</property>

<property name="pageSize" value="#{4 * 1024}" />

<property name="storagePath" value="${grid.data}" />

<property name="walPath" value="${grid.wal}" />

<property name="walArchivePath" value="${grid.wal}"/>

<property name="walMode" value="BACKGROUND" />

<property name="walFlushFrequency" value="10000"></property>

</bean>

</property>

Thanx and Regards,
KR Kumar

Re: number of way segments in wal

[akurbanov <an...@gmail.com>]

Hi,

According to mentioned ticket
(https://issues.apache.org/jira/browse/IGNITE-10840), this is a critical
part of reproducer. Looks like this WAL archive disabling option works for
clean WAL currently.

I would recommend doing following things:

1. Backup the whole work directory.
2. Remove $IGNITE_HOME/work/wal and $IGNITE_HOME/work/db/NODE_UUID/cp
(replace with real node UUID or consistent id if you have set one)
directories and try to restart with the same configuration.

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

["krkumar24061975@gmail.com" <kr...@gmail.com>]

Hi Anton - Initially we had the wal and wal archive configured to different
folders, later we changed the config to the same folder and restarted the
cluster. Is that a problem?

Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

[akurbanov <an...@gmail.com>]

Hello,

One important question: if you can recall, did you start a clean cluster
with WAL /WAL archive pointing to the very same directory, or you have
stopped the node without WAL /PDS cleanup and changed this setting?

So, when this configuration was applied, were there real WAL segment files
in WAL folder?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

["krkumar24061975@gmail.com" <kr...@gmail.com>]

Hi Anton - This is probably not our issue as I am not seeing any Exceptions
in the log

Thnx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

[akurbanov <an...@gmail.com>]

Hi,

Have just found an issue in Ignite JIRA:
https://issues.apache.org/jira/browse/IGNITE-10840

Does it fits your case?

Seems like this point is crucial:

/4)Change the walArchivePath to walPath as it described here:

See config2.xml

https://apacheignite.readme.io/docs/write-ahead-log#section-disabling-wal-archiving/

Was this configuration changed while you had live segments in wal folder?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

["krkumar24061975@gmail.com" <kr...@gmail.com>]

Hi Anton - Thanks a lot and this helps me understand the problem. 

I am still trying to get the logs from production and it might take some
more time.

I did see a message in the logs saying "checkpoint process failed" - What
are the consequences and how should i handle such errors. What are the
reasons for which i could get into this error?? Yes there is a node that
went down and rebalance was happening I guess. Could this create problems
and what sort ?? No actions on baseline or WAL.

As far as data streaming is concerned, data was streamed at 80K events per
second. each event is about 1KB size. And also lot of Ignite SQLs are being
executed (INSERSTS AND UPDATES)

Thanx and Regards,
KR Kumar





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

[akurbanov <an...@gmail.com>]

Hello,

Could you please share the full log? This must shed some light on events
happened prior to the issue. 

I suspect that there is some checkpoint process failure logged, you might
look up for occurences of "Failed to process checkpoint" or "Failed to find
checkpoint record at the given WAL pointer" in logs. Another possibility
that comes to mind is WAL reservation for rebalance.

Could you shortly describe the workload under which you do experience this
issue? Any baseline changes, actions on WAL, what are the data volumes that
are streamed/inserted/removed into caches using specific API.

Specific OS/Java versions would be great as well.

Best regards,
Anton





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

["krkumar24061975@gmail.com" <kr...@gmail.com>]

Hi - I am currently using ignite version 2.7.6 and the files do get deleted
whenever i restart the server but after that they continuously stack-up. One
thing that i have noticed in the log files is this message "Could not clear
historyMap due to WAL reservation on cp:". I checked the code and found that
its either file is locked or reserved( not sure what is reserved). Any
reason why the wal files are getting locked up or reserved. 

Let me know if you need any information. 

Thanx and Regards,
KR Kumar 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: number of way segments in wal

[akurbanov <an...@gmail.com>]

Hello,

In the case with WAL archive turned off, there might be more that 10 files
in /wal folder. It doesn't acts like a ring buffer in this case, the
segments are just created in the same folder, but the size should be
controlled by DataStorageConfiguration.maxWalArchiveSize (defaults to 4x
checkpoint buffer) in latest version.

Could you please tell the exact Ignite version where you do see this issue?
And have you noticed, had WAL segments ever been deleted, or they are
stacking from the very first segment?

Do you have the logs for this case, how often checkpoints are created?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/