You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Thomas Weise <th...@apache.org> on 2016/02/21 19:09:00 UTC
Integration with Apache Apex (incubating)
Dear Ignite community,
Apache Apex (http://apex.incubator.apache.org/) is an in-memory stream
processing and analytics platform, built for scalability, low-latency
processing, high availability and operability. It has a pipeline processing
architecture can be used for real-time and batch processing. Apex runs
natively on YARN and HDFS.
Typical Apex applications integrate with external systems: ingestion, state
saving, lookup data for processing, land compute results for downstream
consumers etc. The Malhar library has a wide range of connectors and
compute functions readily available to assemble Apex applications.
I wanted to see if there is interest to develop Ignite integrations as part
of Malhar. Some ideas:
- Checkpointing (
https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/) could
use the IGFS (possibly with Ignite native integration of the storage agent
interface).
- Compute results could be stored in Ignite (example dimension
computation
https://www.datatorrent.com/blog/blog-dimensions-computation-aggregate-navigator-part-1-intro/
)
- Enrichment/lookup could be implemented with Ignite as storage layer.
Thanks,
Thomas
Re: Integration with Apache Apex (incubating)
Posted by Vladisav Jelisavcic <vl...@gmail.com>.
Hi,
I just recently started working on this,
you can track progress here [1] and of course here [2],
we also started a separate thread here [3].
Best regards,
Vladisav
[1] https://issues.apache.org/jira/browse/IGNITE-3131
[2] https://issues.apache.org/jira/browse/APEXMALHAR-2091
[3]
https://mail-archives.apache.org/mod_mbox/ignite-dev/201605.mbox/%3CCA+5xAo3jOuFk-NDbNrfDw-BckDC9ufRVmYW3x3FOZ=B771GQAg@mail.gmail.com%3E
On Tue, May 24, 2016 at 1:33 AM, sbiradar1975 <sb...@gmail.com> wrote:
> Hello D,
> I landed on this thread while searching for the Apache Ignite integration
> with Apex. Is there an update on this topic somewhere that I might missing?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Integration-with-Apache-Apex-incubating-tp7535p9069.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>
Re: Integration with Apache Apex (incubating)
Posted by sbiradar1975 <sb...@gmail.com>.
Hello D,
I landed on this thread while searching for the Apache Ignite integration
with Apex. Is there an update on this topic somewhere that I might missing?
Thanks!
--
View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Integration-with-Apache-Apex-incubating-tp7535p9069.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Integration with Apache Apex (incubating)
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Hi Thomas,
Welcome to the Ignite community! Thanks for suggesting various ways to
integrate with Apache Apex. Give me a couple of days to look at it deeper
and respond.
D.
On Sun, Feb 21, 2016 at 10:09 AM, Thomas Weise <th...@apache.org> wrote:
> Dear Ignite community,
>
> Apache Apex (http://apex.incubator.apache.org/) is an in-memory stream
> processing and analytics platform, built for scalability, low-latency
> processing, high availability and operability. It has a pipeline processing
> architecture can be used for real-time and batch processing. Apex runs
> natively on YARN and HDFS.
>
> Typical Apex applications integrate with external systems: ingestion, state
> saving, lookup data for processing, land compute results for downstream
> consumers etc. The Malhar library has a wide range of connectors and
> compute functions readily available to assemble Apex applications.
>
> I wanted to see if there is interest to develop Ignite integrations as part
> of Malhar. Some ideas:
>
> - Checkpointing (
> https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/)
> could
> use the IGFS (possibly with Ignite native integration of the storage
> agent
> interface).
> - Compute results could be stored in Ignite (example dimension
> computation
>
> https://www.datatorrent.com/blog/blog-dimensions-computation-aggregate-navigator-part-1-intro/
> )
> - Enrichment/lookup could be implemented with Ignite as storage layer.
>
> Thanks,
> Thomas
>