Ignite Stream Processing?

[Matt <dr...@gmail.com>]

Hi all,

I've been reading Ignite docs but I'm not getting something.

It looks to me that the streaming processing API is rather simple, and even
hacky. For instance, Ignite supports stream windows only as a consequence
of having eviction policies on caches, but it lacks many other features
that we generally see on other stream processing frameworks such as Flink.
For instance, sliding windows (same length, different beginning time [1]),
event time windows (times defined by the data itself, not processing time
[2]) and different transformations (filter, fold, reduce, map, etc [3]).

I may be able to do some of this using things like the eviction policy to
define the length of a window, but isn't it a better idea to use Ignite as
an in-memory data storage, and a fully fledged stream processing framework
on top of it to define the transformations to apply to the data?

My initial idea -which I haven't tried yet- is to use collocation to run a
closure where the data resides (affinity call/run), and use that closure to
execute a Flink pipeline (job) locally on that node, then using a custom
made data source I should be able to plug the data from the local Ignite
cache to the Flink pipeline and back into a cache using an Ignite sink.

That would imply running the Flink job locally and completely disable its
own job distribution support (that way I save time and bandwidth since the
job is executed only on one node: the node which owns the data). Is it what
you would/usually do to process Ignite data as a stream?

Any additional insight regarding stream processing on Ignite is very
welcome!

Best regards,
Matt

[1] https://ci.apache.org/projects/flink/flink-docs-release-
1.2/dev/windows.html#sliding-windows
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/
dev/event_time.html
[3] https://ci.apache.org/projects/flink/flink-docs-release-
1.2/dev/datastream_api.html#datastream-transformations

Re: Ignite Stream Processing?

[vdpyatkov <vl...@gmail.com>]

Hi,

Ignite supports simple continuous mapping API[1] for a stream of data.
I do not understand why you will not can filtered a data on map job or
implement you own interface. But I think, you can got issue with collocation
in this approach.

[1]: https://apacheignite.readme.io/docs/continuous-mapping



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Stream-Processing-tp12180p12241.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.