Effective Data through DataStream

[mimmo_c <mi...@gmail.com>]

Hi,

I would like to understand why the amount of data passed to the ignite
servers via datastream is so high compared to the actual data to be passed.
I try to explain my test. I'm passing 10million of data <long,integer> from
1 client to 4 server ignite. The server have 25gb (-J-Xmx25g) of memory for
node, cache partitioned. The time execution take about 20sec, the throughput
is about 20Mb/s so in 20sec the "dirty" data passed through network is about
400Mb. 

The 10million of data size isn't 400mb, but lower, so someone can explain me
what kind of data is passed through the network? At the end of execution
ignite server has in memory actually 400mb used... 

Thanks.
Mimmo.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Effective Data through DataStream

[Dave Harvey <dh...@jobcase.com>]

When you set the stream receiver,  an instance of its class is created and
serialized, which will also include any class it is nested in.   On each
Data Streamer buffer, the serialized form of that class is sent.   If the
class containing the stream receiver has a pointer that is not
Externalizable, then you will serialize that object graph also.

Ensure that your stream receiver is in its own class, and that you are very
careful about the members of that class.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Effective Data through DataStream

[Ilya Lantukh <il...@gridgain.com>]

Hi,

Ignite DataStreamer protocol has overhead for metadata: 11 bytes per
key-value pair + metadata for the whole batch, which size depends on many
factors. So, 400 MB of raw data that you observe is rather logical.

On Tue, Apr 24, 2018 at 9:42 PM, mimmo_c <mi...@gmail.com> wrote:

> Hi,
>
> Someone could help me?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Ilya

Re: Effective Data through DataStream

[mimmo_c <mi...@gmail.com>]

Hi,

Someone could help me?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Effective Data through DataStream

[Mimmo Celano <mi...@gmail.com>]

Hi,

i need a receiver because i want to increase the value of item if it is
stored in cache, like a word-count example. I agree with you about flush.

Thanks,
Mimmo

Il dom 22 apr 2018, 08:57 begineer <re...@gmail.com> ha scritto:

> I dont understand why you need to define receiver to update value in the
> cache. streamer does is for you. you just need to write something like this
>
> try (IgniteDataStreamer<Long, Integer> streamer =
> ignite.dataStreamer("cacheName")) {
>
>         for(Entry<Long, Integer> entry : maps.entrySet()){
>         stmr.addData(entry.getKey().longValue(),entry.getValue());
>         }
> }
>
> datastreamer will flush the data to cache as soon as try with resources is
> completed. you don't even need to flush manually
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Effective Data through DataStream

[begineer <re...@gmail.com>]

I dont understand why you need to define receiver to update value in the
cache. streamer does is for you. you just need to write something like this

try (IgniteDataStreamer<Long, Integer> streamer =
ignite.dataStreamer("cacheName")) {

	for(Entry<Long, Integer> entry : maps.entrySet()){ 
        stmr.addData(entry.getKey().longValue(),entry.getValue()); 
        } 
}

datastreamer will flush the data to cache as soon as try with resources is
completed. you don't even need to flush manually 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Effective Data through DataStream

[mimmo_c <mi...@gmail.com>]

Hi, 

Thanks for response.
The versione of Ignite is 2.0.
Configuration of Server:

    <bean id="cacheconf"
class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="name" value="default"/>
        <property name="atomicityMode" value="ATOMIC"/>
        <property name="backups" value="0"/>
        <property name="cacheMode" value="PARTITIONED"/>
    </bean>

for all 4 Server.

Code:

try(IgniteDataStreamer<Long, Integer> stmr =
ignite.dataStreamer("default")){
               stmr.allowOverwrite(true); --->//Increase time execution
               stmr.perNodeBufferSize(8192);
               stmr.perNodeParallelOperations(16);
               in = System.currentTimeMillis();
               stmr.receiver(new UpdateWord2()); --->//Function for update
value in cache
               
                System.out.println("Size Map "+y+" : "+maps.size());
                for(Entry<Long, Integer> entry : maps.entrySet()){
                	stmr.addData(entry.getKey().longValue(),entry.getValue());
                }
                stmr.flush();
}

I need to caching large amount of data in a Server of 20 Node, with a total
memory of 500Gb. Monitoring the ram memory of Node Server during passing
data through the network 2 Node stuck with java.lang.OutOfMemoryError
consuming all memory while memory consuming of other node is normal.

Thank you for support.
Mimmo




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Effective Data through DataStream

[aealexsandrov <ae...@gmail.com>]

Hi,

Could you please provide your code example, configuration and used Ignite
version?

Thank you,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/