You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by adipro <ad...@zohocorp.com> on 2020/02/03 06:35:32 UTC
Getting No space left on device exception when persistence is
enabled
Hi,
We are using Apache Ignite's persistence enabled mode. The following is the
only configuration we made.
<bean abstract="true" id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true"/>
<property name="clientMode" value="false"/>
<property name="queryThreadPoolSize" value="16" />
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="writeThrottlingEnabled" value="true"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
</bean>
</property>
</bean>
An exception is thrown when the persistence size reaches to 2.4GB and I saw
the off-heap size growing to the same 2.4GB, when the application is
running. I think 3 GB is maximum amount off-heap can store. The ram is of
16GB and disk storage is having still 100 GB left.
Here is the exception:
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write page
[file=/home/test/Documents/AdityaISearchTest/gridgain-community-8.7.7/work/db/node00-b560cf77-8514-4e92-9fa8-9da13fac0ebe/cache-SQL_PUBLIC_DOCIDS/part-4.bin,
pageId=281492156579876, tag=2]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.write(FilePageStore.java:607)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.writeInternal(FilePageStoreManager.java:535)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$WriteCheckpointPages$1.writePage(GridCacheDatabaseSharedManager.java:4516)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.copyPageForCheckpoint(PageMemoryImpl.java:1296)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.checkpointWritePage(PageMemoryImpl.java:1211)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$WriteCheckpointPages.writePages(GridCacheDatabaseSharedManager.java:4479)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$WriteCheckpointPages.run(GridCacheDatabaseSharedManager.java:4405)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: class
org.apache.ignite.IgniteCheckedException: No space left on device
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.write(AsyncFileIO.java:148)
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO$5.run(AbstractFileIO.java:117)
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.fully(AbstractFileIO.java:53)
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:115)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.write(FilePageStore.java:570)
... 9 more
Caused by: class org.apache.ignite.IgniteCheckedException: No space left on
device
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7324)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:190)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145)
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.write(AsyncFileIO.java:145)
... 13 more
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at
sun.nio.ch.SimpleAsynchronousFileChannelImpl$3.run(SimpleAsynchronousFileChannelImpl.java:374)
... 3 more
Can someone please tell me why it's throwing? Because in production we'd be
getting very big amount of data.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Getting No space left on device exception when persistence is
enabled
Posted by adipro <ad...@zohocorp.com>.
We don't have any user limitations. And currently out of 300GB, we have 115
GB free space left. What's with having 100 GB free space?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Getting No space left on device exception when persistence is
enabled
Posted by adipro <ad...@zohocorp.com>.
The whole machine is getting hung up, when I try to start the node. Every app
I open is saying not enough memory. But when I restart and see I could see
still 115 GB of space free.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Getting No space left on device exception when persistence is enabled
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Are you sure it does not write to /tmp?
Regards,
--
Ilya Kasnacheev
ср, 5 февр. 2020 г. в 10:15, Mikael <mi...@telia.com>:
> Hi!
>
> Well, the message say no space left on device so you are sure the
> persistence/WAL data is stored on the drive where you have 100GB free
> space ? you don't have any user limitations on disk usage or anything
> like that ?
>
> Mikael
>
>
> Den 2020-02-05 kl. 07:18, skrev adipro:
> > Can someone please help?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Getting No space left on device exception when persistence is
enabled
Posted by Mikael <mi...@telia.com>.
Hi!
Well, the message say no space left on device so you are sure the
persistence/WAL data is stored on the drive where you have 100GB free
space ? you don't have any user limitations on disk usage or anything
like that ?
Mikael
Den 2020-02-05 kl. 07:18, skrev adipro:
> Can someone please help?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Getting No space left on device exception when persistence is
enabled
Posted by adipro <ad...@zohocorp.com>.
Can someone please help?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/