Storm topologies compatibility and exactly-once

[Olivier Mallassi <ol...@gmail.com>]

hello all

I was reading the apache flink documentation and was particularly
interested in two things
- compatibility with storm api. AFAIU(i need to do more tests), you can
reuse storm defined topologies and run them on a Flink cluster. I was
wondering if this is "still a bêta" or production ready? I have not looked
at the implementation but I also assume there is a kind of bridge between
the Storm APIs and the Flink internals so that you only exexute the
spout/bolt code and do not inherit the Storm internals, right?
- exactly once semantic. I have to say this is a great feature :). I was
wondering if this semantic is still available when running a Storm defined
topology in a Flink cluster (cf my previous point)

Thanks a lot for your help

Cheers

Olivier

Re: Storm topologies compatibility and exactly-once

[Olivier Mallassi <ol...@gmail.com>]

Thanks for the precision Maximilian.

On Tue, Mar 29, 2016 at 11:27 AM, Maximilian Michels <mx...@apache.org> wrote:

> Hi Olivier,
>
> Regarding the general question, please have a look at the
> documentation:
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/storm_compatibility.html
> Yes, you may reuse your existing spouts/bolts, the Storm runtime is
> exchanged for the Flink runtime.
>
> Exactly-once: Within Flink this works as expected but the spouts/bolts
> are not checkpointed at the moment. That means that state won't be
> restored after a failure of the job. Thus, you end up with only
> at-least-once. We plan to change this very soon.
>
> Let us know how your experiments go.
>
> Cheers,
> Max
>
>
>
> On Sun, Mar 27, 2016 at 11:31 AM, Olivier Mallassi
> <ol...@gmail.com> wrote:
> > hello all
> >
> > I was reading the apache flink documentation and was particularly
> interested
> > in two things
> > - compatibility with storm api. AFAIU(i need to do more tests), you can
> > reuse storm defined topologies and run them on a Flink cluster. I was
> > wondering if this is "still a bêta" or production ready? I have not
> looked
> > at the implementation but I also assume there is a kind of bridge between
> > the Storm APIs and the Flink internals so that you only exexute the
> > spout/bolt code and do not inherit the Storm internals, right?
> > - exactly once semantic. I have to say this is a great feature :). I was
> > wondering if this semantic is still available when running a Storm
> defined
> > topology in a Flink cluster (cf my previous point)
> >
> > Thanks a lot for your help
> >
> > Cheers
> >
> > Olivier
>

Re: Storm topologies compatibility and exactly-once

[Maximilian Michels <mx...@apache.org>]

Hi Olivier,

Regarding the general question, please have a look at the
documentation: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/storm_compatibility.html
Yes, you may reuse your existing spouts/bolts, the Storm runtime is
exchanged for the Flink runtime.

Exactly-once: Within Flink this works as expected but the spouts/bolts
are not checkpointed at the moment. That means that state won't be
restored after a failure of the job. Thus, you end up with only
at-least-once. We plan to change this very soon.

Let us know how your experiments go.

Cheers,
Max



On Sun, Mar 27, 2016 at 11:31 AM, Olivier Mallassi
<ol...@gmail.com> wrote:
> hello all
>
> I was reading the apache flink documentation and was particularly interested
> in two things
> - compatibility with storm api. AFAIU(i need to do more tests), you can
> reuse storm defined topologies and run them on a Flink cluster. I was
> wondering if this is "still a bêta" or production ready? I have not looked
> at the implementation but I also assume there is a kind of bridge between
> the Storm APIs and the Flink internals so that you only exexute the
> spout/bolt code and do not inherit the Storm internals, right?
> - exactly once semantic. I have to say this is a great feature :). I was
> wondering if this semantic is still available when running a Storm defined
> topology in a Flink cluster (cf my previous point)
>
> Thanks a lot for your help
>
> Cheers
>
> Olivier