You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by nskovpin <ko...@gmail.com> on 2016/11/28 14:54:28 UTC
Cache transformations
Hello!
My current goal is to transform initial cache to different types of caches.
For example, I have IgniteCache<Long, BinaryObject> initCache ( BinaryObject
value consists of *4* fields) and i want to create IgniteCache<Long,
BinaryObject> secondCache (BinaryObject value consists of *2* fields). I use
BinaryObject type, because it is impossible to me to predict input scheme.
I've created IgniteRunnable class that suits my prupose (copy fields from
inputCache and insert them to a new cache):
<http://apache-ignite-users.70518.x6.nabble.com/file/n9219/runnable.png>
I run this code on 4 nodes with 2Gb and 4 CPU, input data - 1.3 Gb. But the
execusion is very slow (2 min+).
So, i have 3 questions:
1) How can i improve a speed?
2) How can i load data directly on local server nodes (i think that my
approach makes data transfers between server nodes).
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-transformations-tp9219.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache transformations
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hi,
DataStreamer is the best way to load data to cache (even localLoadCache was
implemented over a streamer).
I just want to say, you should not use a DataStreamer in a thread of Ignite
compute (do it asynchronously). Also you can use one instance of Streamer
by node, if you want to better performance..
On Wed, Nov 30, 2016 at 11:54 AM, nskovpin <ko...@gmail.com> wrote:
> Thanks, your answer solved my speed problem. But what about saving data
> into
> a cache? Is there a good approach to save my entities directly in local
> nodes? I know that ignite has a IgniteCache.localLoadCache() method, but it
> requires a CacheStore (I dont need it).
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Cache-transformations-tp9219p9290.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Cache transformations
Posted by nskovpin <ko...@gmail.com>.
Thanks, your answer solved my speed problem. But what about saving data into
a cache? Is there a good approach to save my entities directly in local
nodes? I know that ignite has a IgniteCache.localLoadCache() method, but it
requires a CacheStore (I dont need it).
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-transformations-tp9219p9290.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache transformations
Posted by vdpyatkov <vl...@gmail.com>.
And take attention: "DataStreamer" and "Ignite Compute" use one thred pool
(public thread pool). Hence you should execute work with streamer
asynchronously (other side you risk to get a thread starvation).
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-transformations-tp9219p9227.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache transformations
Posted by vdpyatkov <vl...@gmail.com>.
Hi,
Now one node of Ignite execute the operation at only a one thread. You
should use one thread by partition, those. scan query[1] with method
"setPartition" (instance of local entries).
Using affinity[2] for determine partition by nodes and run your task by
each.
You can slightly increase performance if you are explicitly implement
Externalizable interface for the IgniteRunable.
[1]: http://apacheignite.gridgain.org/docs/cache-queries#scan-queries
[2]:
https://ignite.apache.org/releases/1.7.0/javadoc/org/apache/ignite/Ignite.html#affinity(java.lang.String)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-transformations-tp9219p9226.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.