You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by pragmaticbigdata <am...@gmail.com> on 2017/03/10 14:27:27 UTC
Understanding faster batch inserts/updates with apache ignite 1.9
The latest release
<http://finance.yahoo.com/news/gridgain-professional-edition-1-9-130000667.html>
of ignite introduces streaming mode that enables DML execution operations in
specific scenarios, such as *batch inserts and updates or data preloading*.
I tried following the jira issue
<https://issues.apache.org/jira/browse/IGNITE-4169> but I couldn't get
much.
Talking about batch inserts and updates, as I understand till now, apache
ignite allowed them through igniteCache.putAll() api. Data preloading could
be done through the IgniteDataStreamer addData api or the
igniteCache.loadCache() api that internally called the
CacheStore.loadCache() implementation.
Could someone explain this performance improvement? What is the streaming
mode? What are the API changes with this?
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding faster batch inserts/updates with apache ignite 1.9
Posted by Denis Magda <dm...@apache.org>.
The performance should be comparable. However, as Val pointed out you need to pay some some fee for DML query parsing but it might be negligible.
—
Denis
> On Mar 15, 2017, at 7:21 AM, pragmaticbigdata <am...@gmail.com> wrote:
>
> Thanks for the reply.
>
> DML also implies query parsing, mapping to Java objects and other preparation steps before IgniteDataStreamer API is called. Thus the performance difference.
>
> Does this mean the data streamer api would be faster in pre-loading or bulk-data as compared to the streaming mode jdbc driver?
>
> Thanks.
>
> On Wed, Mar 15, 2017 at 12:24 AM, vkulichenko [via Apache Ignite Users] <[hidden email] <x-msg://17/user/SendEmail.jtp?type=node&node=11195&i=0>> wrote:
> 1. Yes, it's using IgniteDataStreamer under the hood and even has the same parameters.
> 2. DML also implies query parsing, mapping to Java objects and other preparation steps before IgniteDataStreamer API is called. Thus the performance difference.
>
> -Val
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11170.html <http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11170.html>
> To unsubscribe from Understanding faster batch inserts/updates with apache ignite 1.9, click here <applewebdata://52A97E6D-2B91-449D-8093-7012855ECEC0>.
> NAML <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Understanding faster batch inserts/updates with apache ignite 1.9 <http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11195.html>
> Sent from the Apache Ignite Users mailing list archive <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
Re: Understanding faster batch inserts/updates with apache ignite
1.9
Posted by pragmaticbigdata <am...@gmail.com>.
Thanks for the reply.
DML also implies query parsing, mapping to Java objects and other
> preparation steps before IgniteDataStreamer API is called. Thus the
> performance difference.
Does this mean the data streamer api would be faster in pre-loading or
bulk-data as compared to the streaming mode jdbc driver?
Thanks.
On Wed, Mar 15, 2017 at 12:24 AM, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n11170h54@n6.nabble.com> wrote:
> 1. Yes, it's using IgniteDataStreamer under the hood and even has the same
> parameters.
> 2. DML also implies query parsing, mapping to Java objects and other
> preparation steps before IgniteDataStreamer API is called. Thus the
> performance difference.
>
> -Val
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-
> inserts-updates-with-apache-ignite-1-9-tp11117p11170.html
> To unsubscribe from Understanding faster batch inserts/updates with apache
> ignite 1.9, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11117&code=YW1pdHMuODRAZ21haWwuY29tfDExMTE3fDU1OTcxMzI3Nw==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11195.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding faster batch inserts/updates with apache ignite
1.9
Posted by vkulichenko <va...@gmail.com>.
1. Yes, it's using IgniteDataStreamer under the hood and even has the same
parameters.
2. DML also implies query parsing, mapping to Java objects and other
preparation steps before IgniteDataStreamer API is called. Thus the
performance difference.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11170.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding faster batch inserts/updates with apache ignite
1.9
Posted by pragmaticbigdata <am...@gmail.com>.
Thanks for sharing the link. I have a couple of questions on the streaming
mode support
1. How soon are the cache entries added via the jdbc driver streaming mode
available for other requests like querying? Is it similar to how
IgniteDataStreamer api behaves i.e. the cache entries are available as soon
as the client flushes them to the server on timeout or when buffer size is
full?
2. How is this approach faster for bulk inserts or pre-loading when
comparing it with a client directly using IgniteDataStreamer api ? I am
trying to understand the performance gain over 1.8 which is documented
<http://finance.yahoo.com/news/gridgain-professional-edition-1-9-130000667.html>
as 20%-40%.
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11165.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding faster batch inserts/updates with apache ignite 1.9
Posted by Denis Magda <dm...@gridgain.com>.
Here you can read more about this mode
https://apacheignite.readme.io/docs/jdbc-driver#streaming-mode
On Friday, March 10, 2017, vkulichenko <va...@gmail.com>
wrote:
> This is about JBDC driver and DML updates. IgniteJdbcDriver now has
> streaming
> mode in which all updates will be propagated to the cluster though
> IgniteDataStreamer. Otherwise they will executed separately without
> batching.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Understanding-faster-batch-
> inserts-updates-with-apache-ignite-1-9-tp11117p11126.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Understanding faster batch inserts/updates with apache ignite
1.9
Posted by vkulichenko <va...@gmail.com>.
This is about JBDC driver and DML updates. IgniteJdbcDriver now has streaming
mode in which all updates will be propagated to the cluster though
IgniteDataStreamer. Otherwise they will executed separately without
batching.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-faster-batch-inserts-updates-with-apache-ignite-1-9-tp11117p11126.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.