Data streamers: How to ensure loading data for the current partition only

[Sumanta Ghosh <su...@yahoo.com>]

Hi,
Want to ensure that data streamer in each node only loads the relevant data
for that node (I am assuming that data streamers are running in all nodes
that are up).
In the documentation - in the Cache loader, it mentions about storing
affinity key in DB and then retrieve the data based on affinity key. Will
the same apply for data streamers as well? If yes, what is the good way to
update the affinity key in case there is addition of node in the cluster?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-streamers-How-to-ensure-loading-data-for-the-current-partition-only-tp10949.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Data streamers: How to ensure loading data for the current partition only

[Sumanta Ghosh <su...@yahoo.com>]

Yes - so taking CacheLoader route then. Thanks a lot for clearing this up.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-streamers-How-to-ensure-loading-data-for-the-current-partition-only-tp10949p10996.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Data streamers: How to ensure loading data for the current partition only

[vkulichenko <va...@gmail.com>]

Sumanta,

Streamer is generally used on a client node to stream data into the cluster.
It only can be created by your code, so you have full control on how many
streamers are there.

If you want to pull data directly from server nodes, use
CacheStore.loadCache instead.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-streamers-How-to-ensure-loading-data-for-the-current-partition-only-tp10949p10981.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Data streamers: How to ensure loading data for the current partition only

[Sumanta Ghosh <su...@yahoo.com>]

Hi Val,
Thanks a lot - yes, the problem I mentioned does not exist if the data
streamer run on a single node. However, this raises another question though.
How to ensure that data streamers run on a single node only? Is it done by
ignite automatically? If not, can you suggest any best practice for that?
I am running a single sprint boot application which also starts ignite. In
order to keep the deployment simple, I was trying to avoid a dedicated node
for ignite only. 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-streamers-How-to-ensure-loading-data-for-the-current-partition-only-tp10949p10960.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Data streamers: How to ensure loading data for the current partition only

[vkulichenko <va...@gmail.com>]

IgniteDataStreamer is supposed to be running on a single node loading all the
data. It will automatically map entries to correct nodes, batch them and
stream efficiently.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-streamers-How-to-ensure-loading-data-for-the-current-partition-only-tp10949p10954.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.