[jira] [Commented] (IGNITE-6699) Optimize client-side data streamer performance

["Ivan Fedotov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419515#comment-16419515 ] 

Ivan Fedotov commented on IGNITE-6699:
--------------------------------------

[~NIzhikov], Can you look at my changes, please?

> Optimize client-side data streamer performance
> ----------------------------------------------
>
>                 Key: IGNITE-6699
>                 URL: https://issues.apache.org/jira/browse/IGNITE-6699
>             Project: Ignite
>          Issue Type: Task
>          Components: streaming
>    Affects Versions: 2.3
>            Reporter: Vladimir Ozerov
>            Assignee: Ivan Fedotov
>            Priority: Major
>              Labels: iep-1, performance
>             Fix For: 2.5
>
>
> Currently if a user has several server nodes and a single client node with single thread pushing data to streamer, he will not be able to load data at maximum speed. On the other hand, if he start several data loading threads, throughput will increase. 
> One of root causes of this is bad data streamer design. Method {{IgniteDataStreamer.addData(K, V)}} returns new feature for every operation, this is too fine grained approach. Also it generates a lot of garbage and causes contention on streamer internals. 
> Proposed implementation flow:
> 1) Compare performance of {{addData(K, V)}} vs {{addData(Collection)}} methods from one thread in distributed environment. The latter should show considerably higher throughput.
> 2) Users should receive per-batch features, rather than per-key. 
> 3) Try caching thread data in some collection until it is large enough to avoid contention and unnecessary allocations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)