You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by rishi007bansod <ri...@gmail.com> on 2017/07/03 18:06:06 UTC
Re: Ignite Data Streamer Performance is not improving with increase
in threads
Hi Andrey,
Attached is the code we have used for bench marking. Is there any tuning
that we can apply to get better performance out of ignite single instance
further?
Also we have attached logs taken from our tool where we varied
datastreamer parallelism from 1 to 16(default). In this case it is observed
that,
(1) ignite by default creates threadpool of size 56 and datastreamer uses
threads among this threadool depending upon parallelism set(is it correct??,
correct me if I am wrong)
(2) Also when datastreamer parallelism is set to 1, it is observed that
while loop thread(Timer-0) goes into waiting state after some interval(here
we get 30k rate)(Why is this happening? why rate here is limited to 30k
instead of 80K(80k is rate in case of default parallelism))
(3) Whereas in case of default parallelism(i.e. 16) while loop
thread(Timer-0) is continuously in running state(here we get 80k rate only).
But in this case public thread pools of data streamers are waiting most of
the time, is this the reason for less throughput?
<http://apache-ignite-users.70518.x6.nabble.com/file/n14276/SingleThread.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n14276/DefaultThreads.png>
code.java
<http://apache-ignite-users.70518.x6.nabble.com/file/n14276/code.java>
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Performance-is-not-improving-with-increase-in-threads-tp14151p14276.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Data Streamer Performance is not improving with increase
in threads
Posted by rishi007bansod <ri...@gmail.com>.
Hi Andrew,
We have observed that following method(segment.access()) blocks ignite
data caching using data streamers(For single ignite instance). This limits
our resource utilization i.e. CPU, MEM are not fully utilized. How can we
avoid this blocking so that we can get maximum performance out of single
ignite instance?
<http://apache-ignite-users.70518.x6.nabble.com/file/n14623/blockedthreads.png>
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Performance-is-not-improving-with-increase-in-threads-tp14151p14623.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Data Streamer Performance is not improving with increase
in threads
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Rishi,
1. Yes. Ignite datastreamer uses datastreamer-pool (from 2.0 version) with
size equals to number of available processors.
2. In case of datastreamer parallelism is set to 1, streamer will not send
next batch until it get ack from previous one.
With default settings, streamer can send up to 16 batches per node without
waiting for ack.
3. Seems, Timer-0 thread is a bottleneck.
On Mon, Jul 3, 2017 at 9:06 PM, rishi007bansod <ri...@gmail.com>
wrote:
> Hi Andrey,
> Attached is the code we have used for bench marking. Is there any
> tuning
> that we can apply to get better performance out of ignite single instance
> further?
> Also we have attached logs taken from our tool where we varied
> datastreamer parallelism from 1 to 16(default). In this case it is observed
> that,
> (1) ignite by default creates threadpool of size 56 and datastreamer uses
> threads among this threadool depending upon parallelism set(is it
> correct??,
> correct me if I am wrong)
> (2) Also when datastreamer parallelism is set to 1, it is observed that
> while loop thread(Timer-0) goes into waiting state after some interval(here
> we get 30k rate)(Why is this happening? why rate here is limited to 30k
> instead of 80K(80k is rate in case of default parallelism))
> (3) Whereas in case of default parallelism(i.e. 16) while loop
> thread(Timer-0) is continuously in running state(here we get 80k rate
> only).
> But in this case public thread pools of data streamers are waiting most of
> the time, is this the reason for less throughput?
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n14276/SingleThread.png>
>
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n14276/DefaultThreads.png>
>
> code.java
> <http://apache-ignite-users.70518.x6.nabble.com/file/n14276/code.java>
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Data-Streamer-Performance-is-
> not-improving-with-increase-in-threads-tp14151p14276.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov