You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Edward Chen <ja...@gmail.com> on 2020/07/07 20:18:23 UTC

Ignite server of perNodeParallelOperatoins ?

Hello,

In IgniteDataStreamer, there is a config: perNodeParallelOperatoins 
(int), it is configured in the client side. in the Server side, does it 
have similar configuration ? otherwise,  if client has freedom to set 
any number of perNodeParallelOperatoins they want,  how server prevent 
not crash ?

Thanks. Ed



Re: Ignite server of perNodeParallelOperatoins ?

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Yes, there is Data Streamer Thread Pool Size on server, which corresponds
to perNodeParallelOperations.

Server will not run more parallel operations than thread pool size.

Regards,
-- 
Ilya Kasnacheev


вт, 7 июл. 2020 г. в 23:18, Edward Chen <ja...@gmail.com>:

> Hello,
>
> In IgniteDataStreamer, there is a config: perNodeParallelOperatoins
> (int), it is configured in the client side. in the Server side, does it
> have similar configuration ? otherwise,  if client has freedom to set
> any number of perNodeParallelOperatoins they want,  how server prevent
> not crash ?
>
> Thanks. Ed
>
>
>

Re: Ignite server of perNodeParallelOperatoins ?

Posted by akorensh <al...@gmail.com>.
Hi,
   A dataStreamer collects entries into a buffer, and when it is full, sends
the buffer over to a server node.. see: perNodeBufferSize(int) 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#perNodeBufferSize-int-

   perNodeParallelOperations is a client side setting that determines how
many buffers to send to a given server node without waiting for an
acknowledgement. The server node will try to process and acknowledge as many
buffers as it is given. perNodeParallelOperations is a throttling parameter
to make sure receiver nodes are not overloaded.

   IngiteStreamer.addData will block when the limit specified
perNodeParallelOperations is reached.

see:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#perNodeParallelOperations-int-


from the doc:
  perNodeParallelOperations(int) - sometimes data may be added to the data
streamer via addData(Object, Object) method faster than it can be put in
cache. In this case, new buffered stream messages are sent to remote nodes
before responses from previous ones are received. This could cause unlimited
heap memory utilization growth on local and remote nodes. To control memory
utilization, this setting limits maximum allowed number of parallel buffered
stream messages that are being processed on remote nodes. If this number is
exceeded, then addData(Object, Object) method will block to control memory
utilization. Default is equal to CPU count on remote node multiply by
DFLT_PARALLEL_OPS_MULTIPLIER.

see:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/