Re: Ignite architecture

[dsetrakyan <ds...@apache.org>]

Hi,

Please properly subscribe to the user list, so everyone in the community
could get the notifications. All you need to do is sent an empty email to
"user-subscribe@ignite.apache.org" and follow the instructions in the reply.


andy@bdp wrote
> I need to create an architecture for Ignite that allows:
> 
> Data to be streamed into a cache.

You will likely need to use IgniteDataStreamer API for this.

andy@bdp wrote
> A cache listener to put selected entries in a queue. One or more
> distributed processes to wait for queue entries, do some work and put the
> results back in a cache.

The streamer comes with StreamReceiver which allows you to process the data
before it gets put in a cache. I am not sure what you mean by processing
"waiting" for queue entries. Would it be sufficient to have this processes
implement their logic in the StreamReceiver?

andy@bdp wrote
> Also, point me in the direction of an example(s) of a listener putting
> something on a queue and 1 or more processes consuming the queue entries.

Ignite comes with many examples. I think in your case you should take a look
at the streamer examples here:
https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming
https://github.com/apache/ignite/tree/master/examples/src/main/java8/org/apache/ignite/examples/java8/streaming

andy@bdp wrote
> Does anyone have a sample architecture diagram that Ignite conforms to?

There are many architectural diagrams in the documentation. If you are
asking how the keys are distributed, perhaps you will find this
documentation useful:
https://apacheignite.readme.io/docs/cache-modes



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-architecture-tp1588p1595.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite architecture

["andy@bdp" <an...@bigdatapartnership.com>]

Hi Dmitriy

Thank you, for your reply. I didn't realize you had called into our office,
but then I have been on-site for a long time. I initiated the conversation
with Jim Leonard some time ago and would have liked to have met.

 I have looked through all the examples and am struggling to understand the
appropriate Ignite technology to use.

I am already using the DataStreamer to load files into a cache. I need to
check that a bundle of different files have been cached within a time
limit. I have a listener being triggered on put. So when an xml file
arrives I need to check the cache for the other files in the bundle so they
can be packaged.

I propose the listener puts the xml file key into a queue and have many
distributed processes "taking" from the queue to perform the bundling if
all the files have arrived in the cache. This would become a general
purpose pattern for a machine learning workflow I think this should be a
simple task for Ignite.

Bets wishes,
Andy


Andrew Rendle
Big Data Partnership

Mob: +44 (0)7973 878 454
Tel:   +44 (0)20 7 205 2550

andrew.rendle@bigdatapartnership.com
twitter.com/bigdataexperts
www.bigdatapartnership.com

On 13 October 2015 at 00:14, dsetrakyan [via Apache Ignite Users] <
ml-node+s70518n1595h94@n6.nabble.com> wrote:

> Hi,
>
> Please properly subscribe to the user list, so everyone in the community
> could get the notifications. All you need to do is sent an empty email to "
> user-subscribe@ignite.apache.org" and follow the instructions in the
> reply.
>
> andy@bdp wrote
> I need to create an architecture for Ignite that allows:
>
> Data to be streamed into a cache.
>
> You will likely need to use IgniteDataStreamer API for this.
>
> andy@bdp wrote
> A cache listener to put selected entries in a queue. One or more
> distributed processes to wait for queue entries, do some work and put the
> results back in a cache.
>
> The streamer comes with StreamReceiver which allows you to process the
> data before it gets put in a cache. I am not sure what you mean by
> processing "waiting" for queue entries. Would it be sufficient to have this
> processes implement their logic in the StreamReceiver?
>
> andy@bdp wrote
> Also, point me in the direction of an example(s) of a listener putting
> something on a queue and 1 or more processes consuming the queue entries.
>
> Ignite comes with many examples. I think in your case you should take a
> look at the streamer examples here:
>
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming
>
> https://github.com/apache/ignite/tree/master/examples/src/main/java8/org/apache/ignite/examples/java8/streaming
>
> andy@bdp wrote
> Does anyone have a sample architecture diagram that Ignite conforms to?
>
> There are many architectural diagrams in the documentation. If you are
> asking how the keys are distributed, perhaps you will find this
> documentation useful:
> https://apacheignite.readme.io/docs/cache-modes
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-architecture-tp1588p1595.html
> To unsubscribe from Ignite architecture, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1588&code=YW5kcmV3LnJlbmRsZUBiaWdkYXRhcGFydG5lcnNoaXAuY29tfDE1ODh8MTYzODgzNzc0Ng==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

-- 
 

*NOTICE AND DISCLAIMER*

This email (including attachments) is confidential. If you are not the 
intended recipient, notify the sender immediately, delete this email from 
your system and do not disclose or use for any purpose.

Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United 
Kingdom
Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United 
Kingdom
Big Data Partnership Limited is a company registered in England & Wales 
with Company No 7904824




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-architecture-tp1588p1600.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.