You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Till Rohrmann <tr...@apache.org> on 2019/01/09 15:08:47 UTC

Re: [DISCUSS] Dropping flink-storm?

With https://issues.apache.org/jira/browse/FLINK-10571, we will remove the
Storm topologies from Flink and keep the wrappers for the moment.

However, looking at the FlinkTopologyContext [1], it becomes quite obvious
that Flink's compatibility with Storm is really limited. Almost all of the
context methods are not supported which makes me wonder how useful these
wrappers really are. Given the additional maintenance overhead of having
them in the code base and no indication that someone is actively using
them, I would still be in favour of removing them. This will reduce our
maintenance burden in the future. What do you think?

[1]
https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java

Cheers,
Till

On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com> wrote:

> Yes, let's do it this way.
> The wrapper classes are probably not too complex and can be easily tested.
> We have the same for the Hadoop interfaces, although I think only the
> Input- and OutputFormatWrappers are actually used.
>
>
> Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
> chesnay@apache.org>:
>
>> That sounds very good to me.
>>
>> On 08.10.2018 11:36, Till Rohrmann wrote:
>> > Good point. The initial idea of this thread was to remove the storm
>> > compatibility layer completely.
>> >
>> > During the discussion I realized that it might be useful for our users
>> > to not completely remove it in one go. Instead for those who still
>> > want to use some Bolt and Spout code in Flink, it could be nice to
>> > keep the wrappers. At least, we could remove flink-storm in a more
>> > graceful way by first removing the Topology and client parts and then
>> > the wrappers. What do you think?
>> >
>> > Cheers,
>> > Till
>> >
>> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <chesnay@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> >     I don't believe that to be the consensus. For starters it is
>> >     contradictory; we can't /drop /flink-storm yet still /keep //some
>> >     parts/.
>> >
>> >     From my understanding we drop flink-storm completely, and put a
>> >     note in the docs that the bolt/spout wrappers of previous versions
>> >     will continue to work.
>> >
>> >     On 08.10.2018 11:04, Till Rohrmann wrote:
>> >>     Thanks for opening the issue Chesnay. I think the overall
>> >>     consensus is to drop flink-storm and only keep the Bolt and Spout
>> >>     wrappers. Thanks for your feedback!
>> >>
>> >>     Cheers,
>> >>     Till
>> >>
>> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
>> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
>> >>
>> >>         I've created
>> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
>> >>         removing flink-storm.
>> >>
>> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
>> >>         > Hi everyone,
>> >>         >
>> >>         > I would like to discuss how to proceed with Flink's storm
>> >>         compatibility
>> >>         > layer flink-strom.
>> >>         >
>> >>         > While working on removing Flink's legacy mode, I noticed
>> >>         that some parts of
>> >>         > flink-storm rely on the legacy Flink client. In fact, at
>> >>         the moment
>> >>         > flink-storm does not work together with Flink's new
>> distributed
>> >>         > architecture.
>> >>         >
>> >>         > I'm also wondering how many people are actually using
>> >>         Flink's Storm
>> >>         > compatibility layer and whether it would be worth porting it.
>> >>         >
>> >>         > I see two options how to proceed:
>> >>         >
>> >>         > 1) Commit to maintain flink-storm and port it to Flink's
>> >>         new architecture
>> >>         > 2) Drop flink-storm
>> >>         >
>> >>         > I doubt that we can contribute it to Apache Bahir [1],
>> >>         because once we
>> >>         > remove the legacy mode, this module will no longer work
>> >>         with all newer
>> >>         > Flink versions.
>> >>         >
>> >>         > Therefore, I would like to hear your opinion on this and in
>> >>         particular if
>> >>         > you are using or planning to use flink-storm in the future.
>> >>         >
>> >>         > [1] https://github.com/apache/bahir-flink
>> >>         >
>> >>         > Cheers,
>> >>         > Till
>> >>         >
>> >>
>> >
>>
>>

Re: [DISCUSS] Dropping flink-storm?

Posted by Fabian Hueske <fh...@gmail.com>.
+1 from my side as well.

I would assume that most Bolts that are supported by our current wrappers
can be easily converted into respective Flink functions.

Fabian



Am Do., 10. Jan. 2019 um 10:35 Uhr schrieb Kostas Kloudas <
k.kloudas@da-platform.com>:

> +1 to drop as well.
>
> On Thu, Jan 10, 2019 at 10:15 AM Ufuk Celebi <uc...@apache.org> wrote:
>
>> +1 to drop.
>>
>> I totally agree with your reasoning. I like that we tried to keep it,
>> but I don't think the maintenance overhead would be justified.
>>
>> – Ufuk
>>
>> On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>> >
>> > With https://issues.apache.org/jira/browse/FLINK-10571, we will remove
>> the
>> > Storm topologies from Flink and keep the wrappers for the moment.
>> >
>> > However, looking at the FlinkTopologyContext [1], it becomes quite
>> obvious
>> > that Flink's compatibility with Storm is really limited. Almost all of
>> the
>> > context methods are not supported which makes me wonder how useful these
>> > wrappers really are. Given the additional maintenance overhead of having
>> > them in the code base and no indication that someone is actively using
>> > them, I would still be in favour of removing them. This will reduce our
>> > maintenance burden in the future. What do you think?
>> >
>> > [1]
>> >
>> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com>
>> wrote:
>> >
>> > > Yes, let's do it this way.
>> > > The wrapper classes are probably not too complex and can be easily
>> tested.
>> > > We have the same for the Hadoop interfaces, although I think only the
>> > > Input- and OutputFormatWrappers are actually used.
>> > >
>> > >
>> > > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
>> > > chesnay@apache.org>:
>> > >
>> > >> That sounds very good to me.
>> > >>
>> > >> On 08.10.2018 11:36, Till Rohrmann wrote:
>> > >> > Good point. The initial idea of this thread was to remove the storm
>> > >> > compatibility layer completely.
>> > >> >
>> > >> > During the discussion I realized that it might be useful for our
>> users
>> > >> > to not completely remove it in one go. Instead for those who still
>> > >> > want to use some Bolt and Spout code in Flink, it could be nice to
>> > >> > keep the wrappers. At least, we could remove flink-storm in a more
>> > >> > graceful way by first removing the Topology and client parts and
>> then
>> > >> > the wrappers. What do you think?
>> > >> >
>> > >> > Cheers,
>> > >> > Till
>> > >> >
>> > >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <
>> chesnay@apache.org
>> > >> > <ma...@apache.org>> wrote:
>> > >> >
>> > >> >     I don't believe that to be the consensus. For starters it is
>> > >> >     contradictory; we can't /drop /flink-storm yet still /keep
>> //some
>> > >> >     parts/.
>> > >> >
>> > >> >     From my understanding we drop flink-storm completely, and put a
>> > >> >     note in the docs that the bolt/spout wrappers of previous
>> versions
>> > >> >     will continue to work.
>> > >> >
>> > >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
>> > >> >>     Thanks for opening the issue Chesnay. I think the overall
>> > >> >>     consensus is to drop flink-storm and only keep the Bolt and
>> Spout
>> > >> >>     wrappers. Thanks for your feedback!
>> > >> >>
>> > >> >>     Cheers,
>> > >> >>     Till
>> > >> >>
>> > >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
>> > >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
>> > >> >>
>> > >> >>         I've created
>> > >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
>> > >> >>         removing flink-storm.
>> > >> >>
>> > >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
>> > >> >>         > Hi everyone,
>> > >> >>         >
>> > >> >>         > I would like to discuss how to proceed with Flink's
>> storm
>> > >> >>         compatibility
>> > >> >>         > layer flink-strom.
>> > >> >>         >
>> > >> >>         > While working on removing Flink's legacy mode, I noticed
>> > >> >>         that some parts of
>> > >> >>         > flink-storm rely on the legacy Flink client. In fact, at
>> > >> >>         the moment
>> > >> >>         > flink-storm does not work together with Flink's new
>> > >> distributed
>> > >> >>         > architecture.
>> > >> >>         >
>> > >> >>         > I'm also wondering how many people are actually using
>> > >> >>         Flink's Storm
>> > >> >>         > compatibility layer and whether it would be worth
>> porting it.
>> > >> >>         >
>> > >> >>         > I see two options how to proceed:
>> > >> >>         >
>> > >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
>> > >> >>         new architecture
>> > >> >>         > 2) Drop flink-storm
>> > >> >>         >
>> > >> >>         > I doubt that we can contribute it to Apache Bahir [1],
>> > >> >>         because once we
>> > >> >>         > remove the legacy mode, this module will no longer work
>> > >> >>         with all newer
>> > >> >>         > Flink versions.
>> > >> >>         >
>> > >> >>         > Therefore, I would like to hear your opinion on this
>> and in
>> > >> >>         particular if
>> > >> >>         > you are using or planning to use flink-storm in the
>> future.
>> > >> >>         >
>> > >> >>         > [1] https://github.com/apache/bahir-flink
>> > >> >>         >
>> > >> >>         > Cheers,
>> > >> >>         > Till
>> > >> >>         >
>> > >> >>
>> > >> >
>> > >>
>> > >>
>>
>

Re: [DISCUSS] Dropping flink-storm?

Posted by Fabian Hueske <fh...@gmail.com>.
+1 from my side as well.

I would assume that most Bolts that are supported by our current wrappers
can be easily converted into respective Flink functions.

Fabian



Am Do., 10. Jan. 2019 um 10:35 Uhr schrieb Kostas Kloudas <
k.kloudas@da-platform.com>:

> +1 to drop as well.
>
> On Thu, Jan 10, 2019 at 10:15 AM Ufuk Celebi <uc...@apache.org> wrote:
>
>> +1 to drop.
>>
>> I totally agree with your reasoning. I like that we tried to keep it,
>> but I don't think the maintenance overhead would be justified.
>>
>> – Ufuk
>>
>> On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>> >
>> > With https://issues.apache.org/jira/browse/FLINK-10571, we will remove
>> the
>> > Storm topologies from Flink and keep the wrappers for the moment.
>> >
>> > However, looking at the FlinkTopologyContext [1], it becomes quite
>> obvious
>> > that Flink's compatibility with Storm is really limited. Almost all of
>> the
>> > context methods are not supported which makes me wonder how useful these
>> > wrappers really are. Given the additional maintenance overhead of having
>> > them in the code base and no indication that someone is actively using
>> > them, I would still be in favour of removing them. This will reduce our
>> > maintenance burden in the future. What do you think?
>> >
>> > [1]
>> >
>> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com>
>> wrote:
>> >
>> > > Yes, let's do it this way.
>> > > The wrapper classes are probably not too complex and can be easily
>> tested.
>> > > We have the same for the Hadoop interfaces, although I think only the
>> > > Input- and OutputFormatWrappers are actually used.
>> > >
>> > >
>> > > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
>> > > chesnay@apache.org>:
>> > >
>> > >> That sounds very good to me.
>> > >>
>> > >> On 08.10.2018 11:36, Till Rohrmann wrote:
>> > >> > Good point. The initial idea of this thread was to remove the storm
>> > >> > compatibility layer completely.
>> > >> >
>> > >> > During the discussion I realized that it might be useful for our
>> users
>> > >> > to not completely remove it in one go. Instead for those who still
>> > >> > want to use some Bolt and Spout code in Flink, it could be nice to
>> > >> > keep the wrappers. At least, we could remove flink-storm in a more
>> > >> > graceful way by first removing the Topology and client parts and
>> then
>> > >> > the wrappers. What do you think?
>> > >> >
>> > >> > Cheers,
>> > >> > Till
>> > >> >
>> > >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <
>> chesnay@apache.org
>> > >> > <ma...@apache.org>> wrote:
>> > >> >
>> > >> >     I don't believe that to be the consensus. For starters it is
>> > >> >     contradictory; we can't /drop /flink-storm yet still /keep
>> //some
>> > >> >     parts/.
>> > >> >
>> > >> >     From my understanding we drop flink-storm completely, and put a
>> > >> >     note in the docs that the bolt/spout wrappers of previous
>> versions
>> > >> >     will continue to work.
>> > >> >
>> > >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
>> > >> >>     Thanks for opening the issue Chesnay. I think the overall
>> > >> >>     consensus is to drop flink-storm and only keep the Bolt and
>> Spout
>> > >> >>     wrappers. Thanks for your feedback!
>> > >> >>
>> > >> >>     Cheers,
>> > >> >>     Till
>> > >> >>
>> > >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
>> > >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
>> > >> >>
>> > >> >>         I've created
>> > >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
>> > >> >>         removing flink-storm.
>> > >> >>
>> > >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
>> > >> >>         > Hi everyone,
>> > >> >>         >
>> > >> >>         > I would like to discuss how to proceed with Flink's
>> storm
>> > >> >>         compatibility
>> > >> >>         > layer flink-strom.
>> > >> >>         >
>> > >> >>         > While working on removing Flink's legacy mode, I noticed
>> > >> >>         that some parts of
>> > >> >>         > flink-storm rely on the legacy Flink client. In fact, at
>> > >> >>         the moment
>> > >> >>         > flink-storm does not work together with Flink's new
>> > >> distributed
>> > >> >>         > architecture.
>> > >> >>         >
>> > >> >>         > I'm also wondering how many people are actually using
>> > >> >>         Flink's Storm
>> > >> >>         > compatibility layer and whether it would be worth
>> porting it.
>> > >> >>         >
>> > >> >>         > I see two options how to proceed:
>> > >> >>         >
>> > >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
>> > >> >>         new architecture
>> > >> >>         > 2) Drop flink-storm
>> > >> >>         >
>> > >> >>         > I doubt that we can contribute it to Apache Bahir [1],
>> > >> >>         because once we
>> > >> >>         > remove the legacy mode, this module will no longer work
>> > >> >>         with all newer
>> > >> >>         > Flink versions.
>> > >> >>         >
>> > >> >>         > Therefore, I would like to hear your opinion on this
>> and in
>> > >> >>         particular if
>> > >> >>         > you are using or planning to use flink-storm in the
>> future.
>> > >> >>         >
>> > >> >>         > [1] https://github.com/apache/bahir-flink
>> > >> >>         >
>> > >> >>         > Cheers,
>> > >> >>         > Till
>> > >> >>         >
>> > >> >>
>> > >> >
>> > >>
>> > >>
>>
>

Re: [DISCUSS] Dropping flink-storm?

Posted by Kostas Kloudas <k....@da-platform.com>.
+1 to drop as well.

On Thu, Jan 10, 2019 at 10:15 AM Ufuk Celebi <uc...@apache.org> wrote:

> +1 to drop.
>
> I totally agree with your reasoning. I like that we tried to keep it,
> but I don't think the maintenance overhead would be justified.
>
> – Ufuk
>
> On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org> wrote:
> >
> > With https://issues.apache.org/jira/browse/FLINK-10571, we will remove
> the
> > Storm topologies from Flink and keep the wrappers for the moment.
> >
> > However, looking at the FlinkTopologyContext [1], it becomes quite
> obvious
> > that Flink's compatibility with Storm is really limited. Almost all of
> the
> > context methods are not supported which makes me wonder how useful these
> > wrappers really are. Given the additional maintenance overhead of having
> > them in the code base and no indication that someone is actively using
> > them, I would still be in favour of removing them. This will reduce our
> > maintenance burden in the future. What do you think?
> >
> > [1]
> >
> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com> wrote:
> >
> > > Yes, let's do it this way.
> > > The wrapper classes are probably not too complex and can be easily
> tested.
> > > We have the same for the Hadoop interfaces, although I think only the
> > > Input- and OutputFormatWrappers are actually used.
> > >
> > >
> > > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
> > > chesnay@apache.org>:
> > >
> > >> That sounds very good to me.
> > >>
> > >> On 08.10.2018 11:36, Till Rohrmann wrote:
> > >> > Good point. The initial idea of this thread was to remove the storm
> > >> > compatibility layer completely.
> > >> >
> > >> > During the discussion I realized that it might be useful for our
> users
> > >> > to not completely remove it in one go. Instead for those who still
> > >> > want to use some Bolt and Spout code in Flink, it could be nice to
> > >> > keep the wrappers. At least, we could remove flink-storm in a more
> > >> > graceful way by first removing the Topology and client parts and
> then
> > >> > the wrappers. What do you think?
> > >> >
> > >> > Cheers,
> > >> > Till
> > >> >
> > >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <
> chesnay@apache.org
> > >> > <ma...@apache.org>> wrote:
> > >> >
> > >> >     I don't believe that to be the consensus. For starters it is
> > >> >     contradictory; we can't /drop /flink-storm yet still /keep
> //some
> > >> >     parts/.
> > >> >
> > >> >     From my understanding we drop flink-storm completely, and put a
> > >> >     note in the docs that the bolt/spout wrappers of previous
> versions
> > >> >     will continue to work.
> > >> >
> > >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
> > >> >>     Thanks for opening the issue Chesnay. I think the overall
> > >> >>     consensus is to drop flink-storm and only keep the Bolt and
> Spout
> > >> >>     wrappers. Thanks for your feedback!
> > >> >>
> > >> >>     Cheers,
> > >> >>     Till
> > >> >>
> > >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
> > >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
> > >> >>
> > >> >>         I've created
> > >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
> > >> >>         removing flink-storm.
> > >> >>
> > >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
> > >> >>         > Hi everyone,
> > >> >>         >
> > >> >>         > I would like to discuss how to proceed with Flink's storm
> > >> >>         compatibility
> > >> >>         > layer flink-strom.
> > >> >>         >
> > >> >>         > While working on removing Flink's legacy mode, I noticed
> > >> >>         that some parts of
> > >> >>         > flink-storm rely on the legacy Flink client. In fact, at
> > >> >>         the moment
> > >> >>         > flink-storm does not work together with Flink's new
> > >> distributed
> > >> >>         > architecture.
> > >> >>         >
> > >> >>         > I'm also wondering how many people are actually using
> > >> >>         Flink's Storm
> > >> >>         > compatibility layer and whether it would be worth
> porting it.
> > >> >>         >
> > >> >>         > I see two options how to proceed:
> > >> >>         >
> > >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
> > >> >>         new architecture
> > >> >>         > 2) Drop flink-storm
> > >> >>         >
> > >> >>         > I doubt that we can contribute it to Apache Bahir [1],
> > >> >>         because once we
> > >> >>         > remove the legacy mode, this module will no longer work
> > >> >>         with all newer
> > >> >>         > Flink versions.
> > >> >>         >
> > >> >>         > Therefore, I would like to hear your opinion on this and
> in
> > >> >>         particular if
> > >> >>         > you are using or planning to use flink-storm in the
> future.
> > >> >>         >
> > >> >>         > [1] https://github.com/apache/bahir-flink
> > >> >>         >
> > >> >>         > Cheers,
> > >> >>         > Till
> > >> >>         >
> > >> >>
> > >> >
> > >>
> > >>
>

Re: [DISCUSS] Dropping flink-storm?

Posted by Kostas Kloudas <k....@da-platform.com>.
+1 to drop as well.

On Thu, Jan 10, 2019 at 10:15 AM Ufuk Celebi <uc...@apache.org> wrote:

> +1 to drop.
>
> I totally agree with your reasoning. I like that we tried to keep it,
> but I don't think the maintenance overhead would be justified.
>
> – Ufuk
>
> On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org> wrote:
> >
> > With https://issues.apache.org/jira/browse/FLINK-10571, we will remove
> the
> > Storm topologies from Flink and keep the wrappers for the moment.
> >
> > However, looking at the FlinkTopologyContext [1], it becomes quite
> obvious
> > that Flink's compatibility with Storm is really limited. Almost all of
> the
> > context methods are not supported which makes me wonder how useful these
> > wrappers really are. Given the additional maintenance overhead of having
> > them in the code base and no indication that someone is actively using
> > them, I would still be in favour of removing them. This will reduce our
> > maintenance burden in the future. What do you think?
> >
> > [1]
> >
> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com> wrote:
> >
> > > Yes, let's do it this way.
> > > The wrapper classes are probably not too complex and can be easily
> tested.
> > > We have the same for the Hadoop interfaces, although I think only the
> > > Input- and OutputFormatWrappers are actually used.
> > >
> > >
> > > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
> > > chesnay@apache.org>:
> > >
> > >> That sounds very good to me.
> > >>
> > >> On 08.10.2018 11:36, Till Rohrmann wrote:
> > >> > Good point. The initial idea of this thread was to remove the storm
> > >> > compatibility layer completely.
> > >> >
> > >> > During the discussion I realized that it might be useful for our
> users
> > >> > to not completely remove it in one go. Instead for those who still
> > >> > want to use some Bolt and Spout code in Flink, it could be nice to
> > >> > keep the wrappers. At least, we could remove flink-storm in a more
> > >> > graceful way by first removing the Topology and client parts and
> then
> > >> > the wrappers. What do you think?
> > >> >
> > >> > Cheers,
> > >> > Till
> > >> >
> > >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <
> chesnay@apache.org
> > >> > <ma...@apache.org>> wrote:
> > >> >
> > >> >     I don't believe that to be the consensus. For starters it is
> > >> >     contradictory; we can't /drop /flink-storm yet still /keep
> //some
> > >> >     parts/.
> > >> >
> > >> >     From my understanding we drop flink-storm completely, and put a
> > >> >     note in the docs that the bolt/spout wrappers of previous
> versions
> > >> >     will continue to work.
> > >> >
> > >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
> > >> >>     Thanks for opening the issue Chesnay. I think the overall
> > >> >>     consensus is to drop flink-storm and only keep the Bolt and
> Spout
> > >> >>     wrappers. Thanks for your feedback!
> > >> >>
> > >> >>     Cheers,
> > >> >>     Till
> > >> >>
> > >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
> > >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
> > >> >>
> > >> >>         I've created
> > >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
> > >> >>         removing flink-storm.
> > >> >>
> > >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
> > >> >>         > Hi everyone,
> > >> >>         >
> > >> >>         > I would like to discuss how to proceed with Flink's storm
> > >> >>         compatibility
> > >> >>         > layer flink-strom.
> > >> >>         >
> > >> >>         > While working on removing Flink's legacy mode, I noticed
> > >> >>         that some parts of
> > >> >>         > flink-storm rely on the legacy Flink client. In fact, at
> > >> >>         the moment
> > >> >>         > flink-storm does not work together with Flink's new
> > >> distributed
> > >> >>         > architecture.
> > >> >>         >
> > >> >>         > I'm also wondering how many people are actually using
> > >> >>         Flink's Storm
> > >> >>         > compatibility layer and whether it would be worth
> porting it.
> > >> >>         >
> > >> >>         > I see two options how to proceed:
> > >> >>         >
> > >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
> > >> >>         new architecture
> > >> >>         > 2) Drop flink-storm
> > >> >>         >
> > >> >>         > I doubt that we can contribute it to Apache Bahir [1],
> > >> >>         because once we
> > >> >>         > remove the legacy mode, this module will no longer work
> > >> >>         with all newer
> > >> >>         > Flink versions.
> > >> >>         >
> > >> >>         > Therefore, I would like to hear your opinion on this and
> in
> > >> >>         particular if
> > >> >>         > you are using or planning to use flink-storm in the
> future.
> > >> >>         >
> > >> >>         > [1] https://github.com/apache/bahir-flink
> > >> >>         >
> > >> >>         > Cheers,
> > >> >>         > Till
> > >> >>         >
> > >> >>
> > >> >
> > >>
> > >>
>

Re: [DISCUSS] Dropping flink-storm?

Posted by Ufuk Celebi <uc...@apache.org>.
+1 to drop.

I totally agree with your reasoning. I like that we tried to keep it,
but I don't think the maintenance overhead would be justified.

– Ufuk

On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org> wrote:
>
> With https://issues.apache.org/jira/browse/FLINK-10571, we will remove the
> Storm topologies from Flink and keep the wrappers for the moment.
>
> However, looking at the FlinkTopologyContext [1], it becomes quite obvious
> that Flink's compatibility with Storm is really limited. Almost all of the
> context methods are not supported which makes me wonder how useful these
> wrappers really are. Given the additional maintenance overhead of having
> them in the code base and no indication that someone is actively using
> them, I would still be in favour of removing them. This will reduce our
> maintenance burden in the future. What do you think?
>
> [1]
> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
>
> Cheers,
> Till
>
> On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com> wrote:
>
> > Yes, let's do it this way.
> > The wrapper classes are probably not too complex and can be easily tested.
> > We have the same for the Hadoop interfaces, although I think only the
> > Input- and OutputFormatWrappers are actually used.
> >
> >
> > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
> > chesnay@apache.org>:
> >
> >> That sounds very good to me.
> >>
> >> On 08.10.2018 11:36, Till Rohrmann wrote:
> >> > Good point. The initial idea of this thread was to remove the storm
> >> > compatibility layer completely.
> >> >
> >> > During the discussion I realized that it might be useful for our users
> >> > to not completely remove it in one go. Instead for those who still
> >> > want to use some Bolt and Spout code in Flink, it could be nice to
> >> > keep the wrappers. At least, we could remove flink-storm in a more
> >> > graceful way by first removing the Topology and client parts and then
> >> > the wrappers. What do you think?
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <chesnay@apache.org
> >> > <ma...@apache.org>> wrote:
> >> >
> >> >     I don't believe that to be the consensus. For starters it is
> >> >     contradictory; we can't /drop /flink-storm yet still /keep //some
> >> >     parts/.
> >> >
> >> >     From my understanding we drop flink-storm completely, and put a
> >> >     note in the docs that the bolt/spout wrappers of previous versions
> >> >     will continue to work.
> >> >
> >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
> >> >>     Thanks for opening the issue Chesnay. I think the overall
> >> >>     consensus is to drop flink-storm and only keep the Bolt and Spout
> >> >>     wrappers. Thanks for your feedback!
> >> >>
> >> >>     Cheers,
> >> >>     Till
> >> >>
> >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
> >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
> >> >>
> >> >>         I've created
> >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
> >> >>         removing flink-storm.
> >> >>
> >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
> >> >>         > Hi everyone,
> >> >>         >
> >> >>         > I would like to discuss how to proceed with Flink's storm
> >> >>         compatibility
> >> >>         > layer flink-strom.
> >> >>         >
> >> >>         > While working on removing Flink's legacy mode, I noticed
> >> >>         that some parts of
> >> >>         > flink-storm rely on the legacy Flink client. In fact, at
> >> >>         the moment
> >> >>         > flink-storm does not work together with Flink's new
> >> distributed
> >> >>         > architecture.
> >> >>         >
> >> >>         > I'm also wondering how many people are actually using
> >> >>         Flink's Storm
> >> >>         > compatibility layer and whether it would be worth porting it.
> >> >>         >
> >> >>         > I see two options how to proceed:
> >> >>         >
> >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
> >> >>         new architecture
> >> >>         > 2) Drop flink-storm
> >> >>         >
> >> >>         > I doubt that we can contribute it to Apache Bahir [1],
> >> >>         because once we
> >> >>         > remove the legacy mode, this module will no longer work
> >> >>         with all newer
> >> >>         > Flink versions.
> >> >>         >
> >> >>         > Therefore, I would like to hear your opinion on this and in
> >> >>         particular if
> >> >>         > you are using or planning to use flink-storm in the future.
> >> >>         >
> >> >>         > [1] https://github.com/apache/bahir-flink
> >> >>         >
> >> >>         > Cheers,
> >> >>         > Till
> >> >>         >
> >> >>
> >> >
> >>
> >>

Re: [DISCUSS] Dropping flink-storm?

Posted by Ufuk Celebi <uc...@apache.org>.
+1 to drop.

I totally agree with your reasoning. I like that we tried to keep it,
but I don't think the maintenance overhead would be justified.

– Ufuk

On Wed, Jan 9, 2019 at 4:09 PM Till Rohrmann <tr...@apache.org> wrote:
>
> With https://issues.apache.org/jira/browse/FLINK-10571, we will remove the
> Storm topologies from Flink and keep the wrappers for the moment.
>
> However, looking at the FlinkTopologyContext [1], it becomes quite obvious
> that Flink's compatibility with Storm is really limited. Almost all of the
> context methods are not supported which makes me wonder how useful these
> wrappers really are. Given the additional maintenance overhead of having
> them in the code base and no indication that someone is actively using
> them, I would still be in favour of removing them. This will reduce our
> maintenance burden in the future. What do you think?
>
> [1]
> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/wrappers/FlinkTopologyContext.java
>
> Cheers,
> Till
>
> On Tue, Oct 9, 2018 at 10:08 AM Fabian Hueske <fh...@gmail.com> wrote:
>
> > Yes, let's do it this way.
> > The wrapper classes are probably not too complex and can be easily tested.
> > We have the same for the Hadoop interfaces, although I think only the
> > Input- and OutputFormatWrappers are actually used.
> >
> >
> > Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <
> > chesnay@apache.org>:
> >
> >> That sounds very good to me.
> >>
> >> On 08.10.2018 11:36, Till Rohrmann wrote:
> >> > Good point. The initial idea of this thread was to remove the storm
> >> > compatibility layer completely.
> >> >
> >> > During the discussion I realized that it might be useful for our users
> >> > to not completely remove it in one go. Instead for those who still
> >> > want to use some Bolt and Spout code in Flink, it could be nice to
> >> > keep the wrappers. At least, we could remove flink-storm in a more
> >> > graceful way by first removing the Topology and client parts and then
> >> > the wrappers. What do you think?
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Mon, Oct 8, 2018 at 11:13 AM Chesnay Schepler <chesnay@apache.org
> >> > <ma...@apache.org>> wrote:
> >> >
> >> >     I don't believe that to be the consensus. For starters it is
> >> >     contradictory; we can't /drop /flink-storm yet still /keep //some
> >> >     parts/.
> >> >
> >> >     From my understanding we drop flink-storm completely, and put a
> >> >     note in the docs that the bolt/spout wrappers of previous versions
> >> >     will continue to work.
> >> >
> >> >     On 08.10.2018 11:04, Till Rohrmann wrote:
> >> >>     Thanks for opening the issue Chesnay. I think the overall
> >> >>     consensus is to drop flink-storm and only keep the Bolt and Spout
> >> >>     wrappers. Thanks for your feedback!
> >> >>
> >> >>     Cheers,
> >> >>     Till
> >> >>
> >> >>     On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler
> >> >>     <chesnay@apache.org <ma...@apache.org>> wrote:
> >> >>
> >> >>         I've created
> >> >>         https://issues.apache.org/jira/browse/FLINK-10509 for
> >> >>         removing flink-storm.
> >> >>
> >> >>         On 28.09.2018 15:22, Till Rohrmann wrote:
> >> >>         > Hi everyone,
> >> >>         >
> >> >>         > I would like to discuss how to proceed with Flink's storm
> >> >>         compatibility
> >> >>         > layer flink-strom.
> >> >>         >
> >> >>         > While working on removing Flink's legacy mode, I noticed
> >> >>         that some parts of
> >> >>         > flink-storm rely on the legacy Flink client. In fact, at
> >> >>         the moment
> >> >>         > flink-storm does not work together with Flink's new
> >> distributed
> >> >>         > architecture.
> >> >>         >
> >> >>         > I'm also wondering how many people are actually using
> >> >>         Flink's Storm
> >> >>         > compatibility layer and whether it would be worth porting it.
> >> >>         >
> >> >>         > I see two options how to proceed:
> >> >>         >
> >> >>         > 1) Commit to maintain flink-storm and port it to Flink's
> >> >>         new architecture
> >> >>         > 2) Drop flink-storm
> >> >>         >
> >> >>         > I doubt that we can contribute it to Apache Bahir [1],
> >> >>         because once we
> >> >>         > remove the legacy mode, this module will no longer work
> >> >>         with all newer
> >> >>         > Flink versions.
> >> >>         >
> >> >>         > Therefore, I would like to hear your opinion on this and in
> >> >>         particular if
> >> >>         > you are using or planning to use flink-storm in the future.
> >> >>         >
> >> >>         > [1] https://github.com/apache/bahir-flink
> >> >>         >
> >> >>         > Cheers,
> >> >>         > Till
> >> >>         >
> >> >>
> >> >
> >>
> >>