You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Roman Guseinov <ro...@gromtech.ru> on 2018/02/15 13:52:12 UTC
Batch size parameter at DataStreamerCacheUpdaters.batched()
Hello Igniters,
In some cases, batched stream receiver can help us to improve performance:
try (IgniteDataStreamer<Integer, String> streamer =
ignite.dataStreamer(cacheName)) {
streamer.receiver(DataStreamerCacheUpdaters.batched());
streamer.addData(getData());
}
Unfortunately, the bad thing is that the receiver internally calls "putAll"
for all data. I think it would be useful to have an option to specify a
batch size like:
DataStreamerCacheUpdaters.batched(256)
What do you think about this? Is it make sense to create a ticket?
Thanks.
Best Regards,
Roman
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Batch size parameter at DataStreamerCacheUpdaters.batched()
Posted by Valentin Kulichenko <va...@gmail.com>.
As far as I remember, it used to be public and then was moved to internal.
The main issue with these updaters was that batching is dangerous because
you can get deadlocks if keys are not sorted (which is the case for
BATCHED). There is also BATCHED_SORTED, but it requires keys to be
Comparable and I doubt we do any validation, so usability is questionable.
We can think about how to improve this functionality, but in current state
I would definitely not include it on public API. User can always implement
custom receiver if needed.
-Val
On Fri, Feb 16, 2018 at 5:04 AM, Denis Mekhanikov <dm...@gmail.com>
wrote:
> Guys,
>
> I think, it makes sense to move this receiver implementation to public API.
> It has much better performance, than the default one, that performs single
> puts.
>
> I don't see any point in adding a batch size parameter to it though, since
> DataStreamer already has such setting itself.
>
> Denis
>
> пт, 16 февр. 2018 г. в 13:29, Roman Guseinov <ro...@gromtech.ru>:
>
> > Val,
> >
> > I got what do you mean. Everything inside the org.apache.ignite.internal
> is
> > considered to be not public API.
> >
> > Best Regards,
> > Roman
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
Re: Batch size parameter at DataStreamerCacheUpdaters.batched()
Posted by Denis Mekhanikov <dm...@gmail.com>.
Guys,
I think, it makes sense to move this receiver implementation to public API.
It has much better performance, than the default one, that performs single
puts.
I don't see any point in adding a batch size parameter to it though, since
DataStreamer already has such setting itself.
Denis
пт, 16 февр. 2018 г. в 13:29, Roman Guseinov <ro...@gromtech.ru>:
> Val,
>
> I got what do you mean. Everything inside the org.apache.ignite.internal is
> considered to be not public API.
>
> Best Regards,
> Roman
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Batch size parameter at DataStreamerCacheUpdaters.batched()
Posted by Roman Guseinov <ro...@gromtech.ru>.
Val,
I got what do you mean. Everything inside the org.apache.ignite.internal is
considered to be not public API.
Best Regards,
Roman
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Batch size parameter at DataStreamerCacheUpdaters.batched()
Posted by Roman Guseinov <ro...@gromtech.ru>.
Hi Val,
Thanks for your response. It seems that is public:
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/datastreamer/DataStreamerCacheUpdaters.java
Best Regards,
Roman
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Batch size parameter at DataStreamerCacheUpdaters.batched()
Posted by Valentin Kulichenko <va...@gmail.com>.
Roman,
DataStreamerCacheUpdaters class is actually not a part of public API, so I
don't see a reason to change it unless there is a need for this internally
in Ignite.
-Val
On Thu, Feb 15, 2018 at 5:52 AM, Roman Guseinov <ro...@gromtech.ru> wrote:
> Hello Igniters,
>
> In some cases, batched stream receiver can help us to improve performance:
>
> try (IgniteDataStreamer<Integer, String> streamer =
> ignite.dataStreamer(cacheName)) {
> streamer.receiver(DataStreamerCacheUpdaters.batched());
>
> streamer.addData(getData());
> }
>
> Unfortunately, the bad thing is that the receiver internally calls "putAll"
> for all data. I think it would be useful to have an option to specify a
> batch size like:
>
> DataStreamerCacheUpdaters.batched(256)
>
> What do you think about this? Is it make sense to create a ticket?
>
> Thanks.
>
> Best Regards,
> Roman
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>