Request for clarity/direction on certain storm features from commiters

[David Smith <da...@gmail.com>]

Hello,

I would like some clarity from committers about certain features and your
intentions about their future direction going forward.

1. Transactional topologie / Linear DRPC Topologies. For the longest time,
this has been informally described as deprecated. But the codebase for this
is still not marked as deprecated. Can we have some guidance on whether
this code will live on and if for how long and when it should be preferred
over Trident (what Trident cannot accomplish that this can)? If it's not
intended to live on, then may I suggested that you deprecate the code
initially and mark it for removal during subsequent release?

2. Trident. There's a lot of good documentation on how to write a Trident
topology but there's hardly any documentation on how to write a Trident
spout. People waste a lot of time attempting to write a Trident spout for
their data sources and give up because it's unclear and all their
assumptions they will bring forward from their knowledge of standard Storm
Spouts are not valid for Trident Spouts. Bringing the Kafka spout into
storm project will address some of these concerns as Trident is fully
supported under Kafka. But it will be useful to know the operational
characteristics that need to be supported by the data source in order to
support a Trident spout. It would be quite helpful to see a simple
implementation of a Trident Spout to serve as guidance for writing a
Trident spout.

3. Trident currently supports building Trident spout based on an existing
IRichSpout's with the understanding that they will only be
non-transactional within Trident. But lot of IRichSpouts cannot be used in
this manner as they expect to receive ack's and fail's. That IRichSpout
contract is not enforced by Trident (understandably) but it would be much
easier for developers to grok this if there was a common API contract that
IRichSpouts that work with Trident can implement (even if only in a
non-transactional manner).

3. DRPC support in regular Storm without Trident. From what I understand
this is not possible right now without using the deprecated
LinearDRPCTopologyBuilder.

4. Tick Tuple support in Trident

5. Storm internals commonly being used by users to develop features. Like
CoordinatedBolt, RotatingMap, etc. These can be highly valuable and we
might want to consider making them a part of the core API for advanced
users and document how they should be used.

6. IBatchBolt. Can this be used without Transactional Topologies?

7. Different stream groupings other than shuffle, field may require better
documentation before users can understand how to use them.

Once again, this is not a complaint but a plea for clarity about future
direction. I understand that some of these are not currently supported and
it may be while before they will be (if ever). That is fine, but now that
Storm is incubating under Apache, it would be nice to have a roadmap that
indicates the direction that Storm is heading towards.

Thank you,
David

Re: Request for clarity/direction on certain storm features from commiters

[Danijel Schiavuzzi <da...@schiavuzzi.com>]

Regarding (2), I think it would be nice to expand the "Trident spouts"
section of the manual (
http://storm.incubator.apache.org/documentation/Trident-spouts.html), since
Trident internals are practically undocumented (no Javadocs, etc. compared
to Storm itself). For example, there's no documentation on the Coordinator
interface. Also, the new Storm website under Incubator doesn't seem to have
an publicly editable wiki, it would be very useful to have one.



On Mon, Mar 31, 2014 at 2:08 PM, David Smith <da...@gmail.com>wrote:

> Maybe this email got lost among a bunch of commits. But I hope that you
> maybe able to give us some insight where you intend to take Storm.
>
> Thank you,
> Peter
>
>
> On Wed, Mar 26, 2014 at 4:30 PM, David Smith <davidksmith2k@gmail.com
> >wrote:
>
> > Hello,
> >
> > I would like some clarity from committers about certain features and your
> > intentions about their future direction going forward.
> >
> > 1. Transactional topologie / Linear DRPC Topologies. For the longest
> time,
> > this has been informally described as deprecated. But the codebase for
> this
> > is still not marked as deprecated. Can we have some guidance on whether
> > this code will live on and if for how long and when it should be
> preferred
> > over Trident (what Trident cannot accomplish that this can)? If it's not
> > intended to live on, then may I suggested that you deprecate the code
> > initially and mark it for removal during subsequent release?
> >
> > 2. Trident. There's a lot of good documentation on how to write a Trident
> > topology but there's hardly any documentation on how to write a Trident
> > spout. People waste a lot of time attempting to write a Trident spout for
> > their data sources and give up because it's unclear and all their
> > assumptions they will bring forward from their knowledge of standard
> Storm
> > Spouts are not valid for Trident Spouts. Bringing the Kafka spout into
> > storm project will address some of these concerns as Trident is fully
> > supported under Kafka. But it will be useful to know the operational
> > characteristics that need to be supported by the data source in order to
> > support a Trident spout. It would be quite helpful to see a simple
> > implementation of a Trident Spout to serve as guidance for writing a
> > Trident spout.
> >
> > 3. Trident currently supports building Trident spout based on an existing
> > IRichSpout's with the understanding that they will only be
> > non-transactional within Trident. But lot of IRichSpouts cannot be used
> in
> > this manner as they expect to receive ack's and fail's. That IRichSpout
> > contract is not enforced by Trident (understandably) but it would be much
> > easier for developers to grok this if there was a common API contract
> that
> > IRichSpouts that work with Trident can implement (even if only in a
> > non-transactional manner).
> >
> > 3. DRPC support in regular Storm without Trident. From what I understand
> > this is not possible right now without using the deprecated
> > LinearDRPCTopologyBuilder.
> >
> > 4. Tick Tuple support in Trident
> >
> > 5. Storm internals commonly being used by users to develop features. Like
> > CoordinatedBolt, RotatingMap, etc. These can be highly valuable and we
> > might want to consider making them a part of the core API for advanced
> > users and document how they should be used.
> >
> > 6. IBatchBolt. Can this be used without Transactional Topologies?
> >
> > 7. Different stream groupings other than shuffle, field may require
> better
> > documentation before users can understand how to use them.
> >
> > Once again, this is not a complaint but a plea for clarity about future
> > direction. I understand that some of these are not currently supported
> and
> > it may be while before they will be (if ever). That is fine, but now that
> > Storm is incubating under Apache, it would be nice to have a roadmap that
> > indicates the direction that Storm is heading towards.
> >
> > Thank you,
> > David
> >
>



-- 
Danijel Schiavuzzi

E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7

Re: Request for clarity/direction on certain storm features from commiters

[David Smith <da...@gmail.com>]

Maybe this email got lost among a bunch of commits. But I hope that you
maybe able to give us some insight where you intend to take Storm.

Thank you,
Peter


On Wed, Mar 26, 2014 at 4:30 PM, David Smith <da...@gmail.com>wrote:

> Hello,
>
> I would like some clarity from committers about certain features and your
> intentions about their future direction going forward.
>
> 1. Transactional topologie / Linear DRPC Topologies. For the longest time,
> this has been informally described as deprecated. But the codebase for this
> is still not marked as deprecated. Can we have some guidance on whether
> this code will live on and if for how long and when it should be preferred
> over Trident (what Trident cannot accomplish that this can)? If it's not
> intended to live on, then may I suggested that you deprecate the code
> initially and mark it for removal during subsequent release?
>
> 2. Trident. There's a lot of good documentation on how to write a Trident
> topology but there's hardly any documentation on how to write a Trident
> spout. People waste a lot of time attempting to write a Trident spout for
> their data sources and give up because it's unclear and all their
> assumptions they will bring forward from their knowledge of standard Storm
> Spouts are not valid for Trident Spouts. Bringing the Kafka spout into
> storm project will address some of these concerns as Trident is fully
> supported under Kafka. But it will be useful to know the operational
> characteristics that need to be supported by the data source in order to
> support a Trident spout. It would be quite helpful to see a simple
> implementation of a Trident Spout to serve as guidance for writing a
> Trident spout.
>
> 3. Trident currently supports building Trident spout based on an existing
> IRichSpout's with the understanding that they will only be
> non-transactional within Trident. But lot of IRichSpouts cannot be used in
> this manner as they expect to receive ack's and fail's. That IRichSpout
> contract is not enforced by Trident (understandably) but it would be much
> easier for developers to grok this if there was a common API contract that
> IRichSpouts that work with Trident can implement (even if only in a
> non-transactional manner).
>
> 3. DRPC support in regular Storm without Trident. From what I understand
> this is not possible right now without using the deprecated
> LinearDRPCTopologyBuilder.
>
> 4. Tick Tuple support in Trident
>
> 5. Storm internals commonly being used by users to develop features. Like
> CoordinatedBolt, RotatingMap, etc. These can be highly valuable and we
> might want to consider making them a part of the core API for advanced
> users and document how they should be used.
>
> 6. IBatchBolt. Can this be used without Transactional Topologies?
>
> 7. Different stream groupings other than shuffle, field may require better
> documentation before users can understand how to use them.
>
> Once again, this is not a complaint but a plea for clarity about future
> direction. I understand that some of these are not currently supported and
> it may be while before they will be (if ever). That is fine, but now that
> Storm is incubating under Apache, it would be nice to have a roadmap that
> indicates the direction that Storm is heading towards.
>
> Thank you,
> David
>