You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mimmo_c <mi...@gmail.com> on 2017/07/14 15:05:43 UTC
Ignite DataStream Troughput
Hi,
We're testing ignite DataStream troughput with a network of 1Gbps. If we
send a big file with scp we have a throughput of 150 Mb/s.
We have 1 Ignite Server and 1 Ignite Client on version 2.0, in Client we
have an HashMap of 1Million Entry which we send into Server's cache. The
Cache is configured as Partitioned, Atomic, no backup. With a simple
DataStream when we send the HashMap into the cache we have a throughput of
20/25 Mb/s without Stream Transformer. Is this a limit of DataStream?
Here is the code:
try(IgniteDataStreamer<String, Long> stmr =
ignite.dataStreamer("default")){
for(Entry<String, Long> entry : mappa1.entrySet()){
stmr.addData(entry.getKey(),entry.getValue());
}
}
When we add another Server Node the each Server Node receive 11 Mb/s
(effective throughput is the same 20/25 Mb/s)
If we Add a StreamTransformer for update value in cache and send more than 1
Hashmap the throughput is 15 Mb/s and if we add another Server Node the
throughput is the same.
public static class UpdateWord extends StreamTransformer<String, Long> {
@Override
public Object process(MutableEntry<String, Long> e, Object... arg)
throws EntryProcessorException {
Object streamed = arg[0];
Long temp = Long.valueOf(streamed.toString()).longValue();
Long val = e.getValue();
e.setValue(val == null ? temp : val + temp);
return null;
}
}
Is there some limit on DataStream Usage or a bottleneck? How can we use all
network bandwidth?
Maybe some DataStream Configuration?
Thanks,
Mimmo
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite DataStream Troughput
Posted by Mimmo Celano <mi...@gmail.com>.
Hi Mikhail,
Thanks for reply.
I tried with set overwrite false, throughput is inscreased and time has
decreased.
3 parallel DataStream 26s on 1 Server Node.
We need to update the entries if we set overwrite false, so how can we do ?
We need a sort of reduce job on Ignite Server, is there any Ignite
functionality already integreted?
Thanks,
Mimmo
2017-07-18 12:12 GMT+02:00 mcherkasov <mc...@gridgain.com>:
> Hi Mimmo,
>
> overwrite true - significantly reduce upload speed, try to avoid it.
>
> Thanks,
> Mikhail.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918p15048.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite DataStream Troughput
Posted by mcherkasov <mc...@gridgain.com>.
Hi Mimmo,
overwrite true - significantly reduce upload speed, try to avoid it.
Thanks,
Mikhail.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918p15048.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite DataStream Troughput
Posted by mimmo_c <mi...@gmail.com>.
Hi,
Thanks Mikhail.
I tried set up Streamer. In my test with an HashMap of 10000000 entries :
1 Server Node 8 Core 32GB Ram JVM
overwrite true
sizebuffer 1024 19s
sizebuffer 4096 18s
sizebuffer 8000 20s
pernodeparallel() 16
I think it's a reasonable time. When sizebuffer is too high you lost time in
populate Stream.
The real problem is when multiple clients insert in parallel data in cache.
Every Client are same HashMap
1 Server Node 8 Core 32GB Ram JVM - 3 Client in parallel, 2 in 1 Node.
overwrite true
sizebuffer 4096 46s
The whole thing becomes a sequential process. Can you further improve the
time in parallel? Does anything escape me? I think it can do better, I'm
probably wrong something in configuration.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918p14981.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite DataStream Troughput
Posted by Mikhail Cherkasov <mc...@gridgain.com>.
On Fri, Jul 14, 2017 at 6:54 PM, mimmo_c <mi...@gmail.com> wrote:
> Thank you for your quick reply.
>
> I will try to set the Streamer. Have you any suggestion for NodeBuffer
> size?
>
unfortunately, I have no recommendations for this one, only experiments
will help here,
it will be great if you will share your results about this.
> If we set big size Buffer and the stream doesn't fill, it streams out
> auntil
> then end?
>
there's
public void autoFlushFrequency(long autoFlushFreq);
by default it is 0, so it's disabled, it will flush data by time out,
also you can flush data manually after cycle that adds data to streamer.
> Thanks
> Mimmo
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918p14922.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Thanks,
Mikhail.
Re: Ignite DataStream Troughput
Posted by mimmo_c <mi...@gmail.com>.
Thank you for your quick reply.
I will try to set the Streamer. Have you any suggestion for NodeBuffer size?
If we set big size Buffer and the stream doesn't fill, it streams out auntil
then end?
Thanks
Mimmo
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918p14922.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite DataStream Troughput
Posted by Mikhail Cherkasov <mc...@gridgain.com>.
Hi Mimmo,
I think you should play with the following streamer parameters:
public int perNodeBufferSize()
public void perNodeParallelOperations(int parallelOps)
dataStream has a buffer per Node, perNodeBufferSize defines how many items
should be
saved in buffer before send it to server node.
perNodeParallelOperations defines how many buffers can be sent to the node
without acknowledge
that it was processed, it's recommended to set it to server cpu count * 2.
Thanks,
Mikhail.
On Fri, Jul 14, 2017 at 6:05 PM, mimmo_c <mi...@gmail.com> wrote:
> Hi,
>
> We're testing ignite DataStream troughput with a network of 1Gbps. If we
> send a big file with scp we have a throughput of 150 Mb/s.
> We have 1 Ignite Server and 1 Ignite Client on version 2.0, in Client we
> have an HashMap of 1Million Entry which we send into Server's cache. The
> Cache is configured as Partitioned, Atomic, no backup. With a simple
> DataStream when we send the HashMap into the cache we have a throughput of
> 20/25 Mb/s without Stream Transformer. Is this a limit of DataStream?
>
>
> Here is the code:
>
> try(IgniteDataStreamer<String, Long> stmr =
> ignite.dataStreamer("default")){
>
> for(Entry<String, Long> entry : mappa1.entrySet()){
> stmr.addData(entry.getKey(),entry.getValue());
> }
>
> }
>
>
> When we add another Server Node the each Server Node receive 11 Mb/s
> (effective throughput is the same 20/25 Mb/s)
>
> If we Add a StreamTransformer for update value in cache and send more than
> 1
> Hashmap the throughput is 15 Mb/s and if we add another Server Node the
> throughput is the same.
>
>
> public static class UpdateWord extends StreamTransformer<String, Long> {
>
> @Override
> public Object process(MutableEntry<String, Long> e, Object... arg)
> throws EntryProcessorException {
> Object streamed = arg[0];
> Long temp = Long.valueOf(streamed.toString()).longValue();
> Long val = e.getValue();
> e.setValue(val == null ? temp : val + temp);
> return null;
> }
> }
>
>
> Is there some limit on DataStream Usage or a bottleneck? How can we use
> all
> network bandwidth?
> Maybe some DataStream Configuration?
>
> Thanks,
> Mimmo
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Thanks,
Mikhail.