You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ja...@accenture.com on 2015/10/15 18:30:43 UTC
?
Hello Ignite users,
Let me share my exaltation from things you're doing.
Let me explain my problem/challenge - sorry I had not time to deeply dive into prototyping yet and I have to make decision very quickly so I use this way to ask general questions which could help me to decide whether to include Ignite into short list for following prototyping of Poc for our solution.
I really appreciate if anybody could respond to my maybe un-competent queries (Q).
Our challenge:
* We have files "BLOBs" which represents binary vectors.
* We'd like to load them into Ignite data grid as distributed key values c
* We are up to implement capability where the service [cid:image002.png@01D10777.9CD40BA0] (our service layer) is getting binary vector (in the same format as stored in file system -> Ignite data grid) and we need to
o perform "search" in our gallery which vector is most close to vector on the input. To calculate the distance (or probability of match) between 2 vectors we have encapsulated in external lib(jar) which basically expose 1:1 match (vector1, vector2) capability. So we need to perform 1:1 match over all vectors we have stored. As the number of vectors in the galley is huge and number of requests to compare is also impressive - this is tack for memory computing.
* What's more, there is service [cid:image004.png@01D10777.9CD40BA0] which continuously adds new records into our gallery
Can you please confirm that following setup/approach with Ignite would work?
* Will we be able to load distributed data [cid:image006.png@01D10777.9CD40BA0] from file storage into grid nodes in distributed manner?
* Will we be able to from service [cid:image002.png@01D10777.9CD40BA0] trigger on every grid node computation function [cid:image008.png@01D10777.9CD40BA0] which will :
o Go over all data (vectors) which are distributed on the current node [cid:image018.png@01D10777.9CD40BA0] ?
o Call external lib be called from computation function [cid:image008.png@01D10777.9CD40BA0]
o Can we pass parameter to computation function [cid:image008.png@01D10777.9CD40BA0]
o Can computation function be deployed on nodes w/o needing to broadcast it from service (computation trigger) ?
o Is there any support from ignite to run computation function [cid:image008.png@01D10777.9CD40BA0] in parallel on certain node?
* We will be able to update data on computation nodes from service [cid:image004.png@01D10777.9CD40BA0] (add new records), so new requests from service [cid:image002.png@01D10777.9CD40BA0] will be able to process updated gallery of vectors in memory
Thank you so MUCH
jaro
[cid:image020.png@01D10777.9CD40BA0]
Jaroslav Saxa
Accenture - Technology Architecture
Plynarenska 7/C, 829 01 Bratislava
Mobile: +421-904-706113
Email: jaroslav.saxa@accenture.com<ma...@accenture.com>
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com
RE: ?
Posted by ja...@accenture.com.
Hello Ivan
Many thanks for great news. We start to prototype ☺
jaro
From: Ivan Veselovskiy [mailto:iveselovskiy@gridgain.com]
Sent: Thursday, October 15, 2015 7:47 PM
To: user@ignite.apache.org
Subject: Re: ?
Hi, Jaroslav,
In my understanding, with Ignite the answers to all the questions are in general positive.
Possibly one of the most challenging tasks there is to adjust cache affinity function to achieve best data/data & data/compute collocation to gain the best performance.
Documentation on affinity collocation can be found there: https://apacheignite.readme.io/docs/affinity-collocation .
On Thu, Oct 15, 2015 at 7:30 PM, <ja...@accenture.com>> wrote:
Hello Ignite users,
Let me share my exaltation from things you’re doing.
Let me explain my problem/challenge – sorry I had not time to deeply dive into prototyping yet and I have to make decision very quickly so I use this way to ask general questions which could help me to decide whether to include Ignite into short list for following prototyping of Poc for our solution.
I really appreciate if anybody could respond to my maybe un-competent queries (Q).
Our challenge:
• We have files “BLOBs” which represents binary vectors.
• We’d like to load them into Ignite data grid as distributed key values c
• We are up to implement capability where the service [cid:image001.png@01D1082B.B8BE7300] (our service layer) is getting binary vector (in the same format as stored in file system -> Ignite data grid) and we need to
o perform “search” in our gallery which vector is most close to vector on the input. To calculate the distance (or probability of match) between 2 vectors we have encapsulated in external lib(jar) which basically expose 1:1 match (vector1, vector2) capability. So we need to perform 1:1 match over all vectors we have stored. As the number of vectors in the galley is huge and number of requests to compare is also impressive – this is tack for memory computing.
• What’s more, there is service [cid:image002.png@01D1082B.B8BE7300] which continuously adds new records into our gallery
Can you please confirm that following setup/approach with Ignite would work?
• Will we be able to load distributed data [cid:image003.png@01D1082B.B8BE7300] from file storage into grid nodes in distributed manner?
• Will we be able to from service [cid:image001.png@01D1082B.B8BE7300] trigger on every grid node computation function [cid:image004.png@01D1082B.B8BE7300] which will :
o Go over all data (vectors) which are distributed on the current node [cid:image005.png@01D1082B.B8BE7300] ?
o Call external lib be called from computation function [cid:image004.png@01D1082B.B8BE7300]
o Can we pass parameter to computation function [cid:image004.png@01D1082B.B8BE7300]
o Can computation function be deployed on nodes w/o needing to broadcast it from service (computation trigger) ?
o Is there any support from ignite to run computation function [cid:image004.png@01D1082B.B8BE7300] in parallel on certain node?
• We will be able to update data on computation nodes from service [cid:image002.png@01D1082B.B8BE7300] (add new records), so new requests from service [cid:image001.png@01D1082B.B8BE7300] will be able to process updated gallery of vectors in memory
Thank you so MUCH
jaro
[cid:image006.png@01D1082B.B8BE7300]
Jaroslav Saxa
Accenture – Technology Architecture
Plynarenska 7/C, 829 01 Bratislava
Mobile: +421-904-706113
Email: jaroslav.saxa@accenture.com<ma...@accenture.com>
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com<http://www.accenture.com>
Re: ?
Posted by Ivan Veselovskiy <iv...@gridgain.com>.
Hi, Jaroslav,
In my understanding, with Ignite the answers to all the questions are in
general positive.
Possibly one of the most challenging tasks there is to adjust cache
affinity function to achieve best data/data & data/compute collocation to
gain the best performance.
Documentation on affinity collocation can be found there:
https://apacheignite.readme.io/docs/affinity-collocation .
On Thu, Oct 15, 2015 at 7:30 PM, <ja...@accenture.com> wrote:
> Hello Ignite users,
>
>
>
> Let me share my exaltation from things you’re doing.
>
> Let me explain my problem/challenge – sorry I had not time to deeply dive
> into prototyping yet and I have to make decision very quickly so I use this
> way to ask general questions which could help me to decide whether to
> include Ignite into short list for following prototyping of Poc for our
> solution.
>
>
>
> I really appreciate if anybody could respond to my maybe un-competent
> queries (Q).
>
>
>
> Our challenge:
>
>
>
> · We have files “BLOBs” which represents binary vectors.
>
> · We’d like to load them into Ignite data grid as distributed key
> values c
>
> · We are up to implement capability where the service (our service
> layer) is getting binary vector (in the same format as stored in file
> system -> Ignite data grid) and we need to
>
> o perform “search” in our gallery which vector is most close to vector
> on the input. To calculate the distance (or probability of match) between 2
> vectors we have encapsulated in *external lib(jar)* which basically
> expose 1:1 match (vector1, vector2) capability. So we need to perform 1:1
> match over all vectors we have stored. As the number of vectors in the
> galley is huge and number of requests to compare is also impressive – this
> is tack for memory computing.
>
> · What’s more, there is service which continuously adds new records
> into our gallery
>
>
>
> Can you please confirm that following setup/approach with Ignite would
> work?
>
> · Will we be able to load distributed data from file storage into
> grid nodes in distributed manner?
>
> · Will we be able to from service trigger on every grid node
> computation function which will :
>
> o Go over all data (vectors) which are distributed on the current node ?
>
>
> o Call external lib be called from computation function
>
> o Can we pass parameter to computation function
>
> o Can computation function be deployed on nodes w/o needing to
> broadcast it from service (computation trigger) ?
>
> o Is there any support from ignite to run computation function in
> *parallel* on certain node?
>
> · We will be able to update data on computation nodes from service (add
> new records), so new requests from service will be able to process
> updated gallery of vectors in memory
>
>
>
> Thank you so MUCH
>
> jaro
>
>
>
>
>
> *Jaroslav Saxa*
>
> *Accenture – Technology Architecture*
>
> Plynarenska 7/C, 829 01 Bratislava
>
> Mobile: +421-904-706113
>
> Email: jaroslav.saxa@accenture.com
>
>
>
> ------------------------------
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
Re: ?
Posted by vkulichenko <va...@gmail.com>.
Hi Jaroslav,
>From what I see you use case definitely fits Ignite.
As a first step I think you should create a partitioned cache and implement
loading the data from your file-based storage. You can use
IgniteDataStreamer or CacheStore APIs for this [1].
It also sound like you will be heavily using compute capabilities. Please
refer to [2] for information about all available features.
[1] https://apacheignite.readme.io/docs/data-loading
[2] https://apacheignite.readme.io/docs/compute-grid
Also here are my answers to your compute-related questions:
jaroslav.saxa wrote
> Go over all data (vectors) which are distributed on the current node ?
Yes, you can use IgniteCache.localEntries() method.
jaroslav.saxa wrote
> Call external lib be called from computation function
No problem with this. You just need to make sure that the library is
available on classpath of all nodes.
jaroslav.saxa wrote
> Can we pass parameter to computation function
Yes, both forkjoin tasks and closure execution support arguments. See
IgniteCompute.execute(..) and IgniteCompute.apply(..) methods.
jaroslav.saxa wrote
> Can computation function be deployed on nodes w/o needing to broadcast it
> from service (computation trigger) ?
Usually there is no issue with sending a closure to a remote node, because
it's lightweight and you have to send a message anyway to trigger the
execution. But if you want to avoid this, you can deploy a task on server
nodes and trigger the execution by its name.
jaroslav.saxa wrote
> Is there any support from ignite to run computation function in parallel
> on certain node?
You can send a closure to any node and it will be submitted to a thread pool
on that node. On the client side you can either wait for the execution
synchronously or have an asynchronous completion listener.
Let us know if you have more questions.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/-tp1624p1626.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.