You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by aMark <fe...@gmail.com> on 2017/11/22 08:35:01 UTC

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Hi,

I am still getting "Too many open files" issue even after increasing the
limit to ~32K.

Following is my ignite configuration:
I am running 2 nodes in cluster configured with REPLICATION mode and
FULL_SYNC.
I have populated ingnite with around ~30 different caches (Requirement to
have close to ~200 caches at any point of time). 
Each cache has around 1.2 million entries. When each cache is created, it
increases the file handle by around 1050. I could see that for each cache
500+ bin files are getting created IgnitePersistentStore directory.

My system runs out of file handle after ~30 caches have been created in
Ignite. Ignite cluster does not seem to release file handle even after
client which is populating cache is killed.
Reason for this could be as Ignite cluster will keep file handle open in
case it need to read data again from the cache. 

I have couple of questions : 
	1. is there a way I can control the number of open file handle in Ignite ?
Either by increasing the bin file size or with  
	2. If I change the cache mode from Replication to partition, will it help
(I assume so as each node will be having only partial data ) ? 



Below is the exception trace I am getting : 

a.util.OptionalDouble [idHash=1820269203, hash=-668113279, isPresent=true,
value=0.03775351299133245]] class org.apache.ignite.IgniteCheckedException:
Can't open file: part-280.bin
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:397)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:291)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:288)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:273)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1014)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:881)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1231)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3179)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2556)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:1999)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:383)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException:
/opt/Ignite/IgnitePersistentStore/10_63_192_28_10_63_193_37_10_63_194_55_127_0_0_1_47500/cache-1324-LOCKED-2/part-280.bin
(Too many open files)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory.create(RandomAccessFileIOFactory.java:38)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:387)
		
		

Thanks,




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,

It looks like there is a new possibility which allows decreasing the number
of open file handles.
As of v2.2, Apache ignite provides a new concept - Cache Groups [1].
Caches within a single cache group share various internal structures and it
allows to mix data of caches in shared partition files. So, this feature
should decrease the number of open file handles.

[1] https://apacheignite.readme.io/v2.3/docs/cache-groups

Thanks!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by aMark <fe...@gmail.com>.
Thanks for your help.

I will look into pointers suggested by you.

Thanks,




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by "slava.koptilin" <sl...@gmail.com>.
Hi,

I think that most of the open file descriptors relate to partition files.
Perhaps, in terms of performance, this is not a very good way to open/close
files after each read/write.

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by userx <ga...@gmail.com>.
Hi Slava,

As per the problem statement defined by Ashish, Whenever a client finishes
off writing, shouldn't the handles be closed and therefore the ever
increasing number of handles will not be a problem otherwise if the handles
of an earlier client operation are not disposed, the resources are still
being consumed ?

Is it something which has been taken care of post 2.1 ?







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Ashish,

>  is there a way I can control the number of open file handle in Ignite ?
>  Or is there a way to control the bin file size ?* 
If I am not mistaken, there is no way to control the bin file size or number
of open file handles.
The one thing that can be changed is a number of partitions for your cache.
Please see [1] [2]
The default values are the following: 1024 - partitioned cache, 512 -
replicated cache.
I don't think that big number of open file handles is a good reason for
reducing the number of partitions.
Anyway, the main rule is that a number of partitions should be bigger than a
number of the nodes in a cluster in order to get a uniform number of
partitions' assignments across all the cluster nodes.


> If I change the cache mode from Replicated to Partitioned.  Will it help ?
In case of 2 nodes, I think it won't help.

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/RendezvousAffinityFunction.html
[2]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by aMark <fe...@gmail.com>.
Hi,

I am running 2 ignite nodes, each one having on different machine but Ignite
might need some more file handles for the .wal files etc hence you see more
number of file handles required.

I am repeating my original questions ,

        1.In my case it creates so many bin files of ~1MB.  If it is
possible to increase bin size, then number of generated bin files will be
less and less number of file handles should be used by Ignite. *  is there a
way I can control the number of open file handle in Ignite ?  Or is there a
way to control the bin file size ?* 


	2. If I change the cache mode from Replicated to Partitioned.  Will it help
? 

Thanks,
Ashish




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to accept TCP connection. java.net.SocketException: Too many open files

Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,

As I mentioned above, Ignite Native Persistence creates a dedicated file on
the file system for every partition that this node is either a primary or
backup.

Well, I guess that you are running two Ignite nodes on the same physical
machine.
The replicated cache has 512 partitions by default.
In that case, number of file handles is approximately equal - 2 (nodes) * 30
(caches) * 512 (partitions) = 30720

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/