Reduce Movement of data in cluster while data loading in PARTITIONED cache

[rishi007bansod <ri...@gmail.com>]

Hi,
     Is there any way we can reduce movement of data across nodes in cluster
while data loading, so that we can speed up data loading process? By
observation I am getting better data loading rates in cache LOCAL mode as
there is no movement of data across ignite instances. Can we do some setting
in PARTITIONED cache mode, so that in this mode also movement across network
gets reduced?

Thanks. 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reduce-Movement-of-data-in-cluster-while-data-loading-in-PARTITIONED-cache-tp10570.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Reduce Movement of data in cluster while data loading in PARTITIONED cache

[vkulichenko <va...@gmail.com>]

If you're using streamer, there is no movement between nodes, only from the
node that streams and data nodes. It's already properly maps the data to
nodes, so partition aware approach is obviously redundant here.

Comparing local and distributed cache doesn't make much sense. If the
streamer is on the same node where data is, it will definitely be much
faster, but that limits you to a single node.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reduce-Movement-of-data-in-cluster-while-data-loading-in-PARTITIONED-cache-tp10570p10608.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Reduce Movement of data in cluster while data loading in PARTITIONED cache

[rishi007bansod <ri...@gmail.com>]

But in my case I am loading data from kafka and not from database, so i am
loading data using ignite data streamer. What can I do for partition aware
data loading in this case(for ignite data streamer)?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reduce-Movement-of-data-in-cluster-while-data-loading-in-PARTITIONED-cache-tp10570p10603.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Reduce Movement of data in cluster while data loading in PARTITIONED cache

[vkulichenko <va...@gmail.com>]

That's correct. Partition aware data loading implies changing the database
schema, but it's generally the fastest way to load the data.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reduce-Movement-of-data-in-cluster-while-data-loading-in-PARTITIONED-cache-tp10570p10581.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Reduce Movement of data in cluster while data loading in PARTITIONED cache

[davida <da...@live.com>]

One thing described here https://apacheignite.readme.io/docs/data-loading is
'Partition-aware data loading', which basically implies implementation of
some kind of 'data partitioning' in your database so that each node only
pulls a portion of data. Hope that helps.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reduce-Movement-of-data-in-cluster-while-data-loading-in-PARTITIONED-cache-tp10570p10571.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.