Re[2]: [DISCUSS] Retire Heron

[Ron Wilcom <rw...@wilcomweb.net>]

All,
We have employed Heron on a large streaming data project for our 
customer and we have a deep vested interest in its adoption/success.  It 
would be very premature to retire Heron while it has not yet reached a 
full Apache release.  Heron has been solid for us and we've been using 
it for well over a year against the largest of data throughputs - its 
held up great and is the most versatile/configurable/industrial 
streaming engine when using the straight/original topology approach -- 
I'm not a fan of the Streamlets API .. why try to compete there? Heron 
should go for being the best industrial solution!

Whatever the hold up for this project is preventing it from moving out 
from incubation should be prioritized for this developer community.  
Many companies are not going to use an "incubated" Apache open source 
product - but once fully released/available in Apache developers will 
have an easier time convincing their technical and business management 
that they can move to Heron and trust that it will have staying power.  
Here is a list of some tasks that I feel need to be tackled to provide 
an upswell of adoption:

* Get out of "Apache incubation" state to a full release - this has to 
happen before there is real/broad adoption - are we looking for 
'perfect', no, because you'll never get there ... perfection can happen 
over time after more open source adoption!
* Capitalize on the history of this project: That it is derived from the 
Twitter development team that Storm projects can easily convert to Heron 
and why that is important (better performance, deployment versatility, 
backpressure features, etc).
* Where are the original founders of this project (Karthik Ramasamy? now 
at Splunk?) - why do they no longer support this project and can we get 
them involved again to help get this over the finish line?  Need more 
high level advocacy all around to promote Heron.
* The technical barrier to entry with Heron is high, it took me a lot of 
grunt work years ago to figure out the proper way to make it work - so 
it is very important to provide strong/deep documentation, getting 
started documentation, and out-of-the-box multi-language examples
* Provide a script/helm based end-to-end deployment that can be run on a 
developers local machine/VM using well known resource managers like 
Kubernetes (e.g. minikube example) - obviously this requires pulling in 
multiple supporting technologies ..... provide expanded examples for 
large/true clusters.
* Need a better administrative tool and monitoring ability across/within 
the topologies - we have many separated topologies working together 
across the stream - its difficult for a new person on the project to see 
how these link together or how the data flows - ability to easily track 
a tuple through - built in ability to send 'canary' tuples to insure 
throughput
* Explain the best practices for including Heron in a wholistic 
streaming architecture such as using Kafka or Pulsar between smaller 
topologies so the stream has queuing break points during 
backpressure/restarts - how to use local caches (Redis) or the best 
approaches for writing out to database end points from bolts, etc
* Correct the default packing algorithm to the original (maybe this is 
already fixed?) - there was a release a while back where the default 
packing algorithm was changed to create a container per bolt/spout which 
is not a good approach on limited hardware (not everyone has 1000 nodes) 
- the concepts of this need better explanation/understanding
* Update the deployment process such that the Heron Client is part of 
compilation but not required to deploy to Kubernetes - result of the 
Heron Client should be standard container images that can be pushed into 
a container repository to be pushed normally like any other image --- 
this will allow it to be more easily mixed in with standard DevOps (DDS) 
procedures
* Allow for on-the-fly (per-environment) configuration settings at the 
point of build/deploy - currently in the version we use its required to 
rebuild per environment (I think this may be handled in a newer release 
but we've been waiting for a final release to occur that gets past 
incubation?)
* FUTURE: Provide even further configuration for spouts/bolts ... allow 
for more dynamic CPU/Memory allocation, special assignment to CPUs/GPUs, 
simplify VM settings, etc
* FUTURE: Expand on functionality such as elastic scaling (Dhalion) but 
provide this within more ubiquitous resource managers like Kubernetes 
(my understanding is that this only works in Mesos/Aurora?).

Thanks all - keep it up - let me know if I can help (my time is limited 
- maybe documentation?)

Ron


------ Original Message ------
From: "Jim Mantheiy" <jm...@gmail.com>
To: dev@heron.apache.org
Cc: "Sree Vaddi" <sr...@yahoo.com>; 
dev@heron.incubator.apache.org
Sent: 10/16/2020 8:32:07 AM
Subject: Re: [DISCUSS] Retire Heron

>All,
>
>I speak for a few people I work with when I say that heron has a unique
>place in the streaming/analytic space. Have used storm, kafka streams, and
>other frameworks. I feel heron is easily the most simplistic, lowest cost
>to entry framework out there.
>
>Personally, if heron would to expand its k8a capabilities such as
>horizontal pod autoscaler, health checks on bolts,, better dashboard,
>perhaps open trace? Then heron would be a one stop shop for highly
>efficient, scalable, robust streaming solution.
>
>Basically, how can I help?
>
>Thanks
>
>Jim
>
>On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io> wrote:
>
>>  Hi All,
>>
>>  Windham, I agree with everything you said. Most importantly what stood out
>>  to me is the lack of documentation that covers why or how someone would use
>>  Heron.  I agree with Dave, we should try to organize and set some goals for
>>  us to complete within the next few weeks and months.
>>
>>  I don't want to see Heron go.  It's the first big open source project I've
>>  worked on and I'd hate to get all teary eyed over a bunch of code that
>>  retired at the Apache Foundation.
>>
>>  How would everyone feel if we used this github project
>>  https://github.com/apache/incubator-heron/projects/4 to track some tasks?
>>  This way we could give some visibility to people trying to learn what's
>>  going on with the community?
>>
>>  What is left outstanding with our 0.23.0-incubating release?  Let's add
>>  those tasks to the github project above.
>>
>>  On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com> wrote:
>>
>>  > Thanks! That's a lot of helpful information!
>>  >
>>  > Agreed that documentation and examples can be better to lower the barrier
>>  > and be more friendly to new users.
>>  >
>>  > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <wi...@stormeye.io>
>>  > wrote:
>>  >
>>  > > I am new to Apache foundation thing and I want to point out a thing
>>  that,
>>  > > when start trying Heron, i got a very big barrier that the
>>  documentation
>>  > is
>>  > > not good enough to do quick start and good understanding the structure
>>  of
>>  > > Heron. I saw a few people asking about how to launch the demo topology
>>  > with
>>  > > facing some technical issues related to python version and
>>  > configurations.
>>  > >
>>  > > For my point of view, we are using Heron as production for log parsing
>>  > > system, and we see great opportunity of increasing the usage of Heron
>>  > with
>>  > > our growth of business. However, recalling my experience when started
>>  > > looking into Heron, the learning curve isn't too high but still much
>>  > higher
>>  > > than other software or systems. I believe the documentation requires
>>  more
>>  > > improvement to let new users to understand more quickly. Furthermore,
>>  > for a
>>  > > business aspect, I believe Heron requires more use case promotion to
>>  > > people. Many don't know what to do with a piece of software and they
>>  > forget
>>  > > about it after some time. Cross-language support (Java/Python/Lua/C++)
>>  is
>>  > > great for people in different field to start using it, but they can't
>>  > find
>>  > > a blog/article/tutorial/youtube to realise what they can do with it. I
>>  am
>>  > > thinking if we can ask companies to share their experience of using
>>  > Heron,
>>  > > and also some personal to share what they can do or their idea would
>>  help
>>  > > the community growth.
>>  > > Sorry for the long words.
>>  > > Windham Wong
>>  > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
>>  > > Co-Founder, Managing Partner of
>>  > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
>>  > > Email // windham.wong@stormeye.io (
>>  > >
>>  >
>>  https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
>>  > > )
>>  > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
>>  > > Fax // +852_3590_2202 (tel:+852_3590_2202)
>>  > >
>>  > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com> wrote:
>>  > > > Thanks Dave!
>>  > > >
>>  > > > IMO our goal is to have an official release, which has been
>>  > challenging.
>>  > > At
>>  > > > the same time, some kubernetes and python works are going on at
>>  least.
>>  > I
>>  > > > remember the issue we found in the latest release candidate was
>>  Python
>>  > 3
>>  > > > related.
>>  > > >
>>  > > >
>>  > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wa...@apache.org>
>>  wrote:
>>  > > > > It would be helpful to have more discussion about what is happening
>>  > on
>>  > > > > this mailing list.
>>  > > > >
>>  > > > > I’m your last active Mentor and I joined only when it seemed like
>>  the
>>  > > > > start of incubation was blocked.
>>  > > > >
>>  > > > > Please show the activity with some visible direction.
>>  > > > >
>>  > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
>>  sree_at_chess@yahoo.com
>>  > > .INVALID>
>>  > > > > wrote:
>>  > > > > >
>>  > > > > > Heron will continue to live long.
>>  > > > > > It has it's own place in the stream processing world among other
>>  > > > > competing technologies.The ever increasing data has stretched
>>  > > competitions
>>  > > > > to the limits of breaking.
>>  > > > > >
>>  > > > > > In addition:
>>  > > > > > In production at the creating company and others around the
>>  > > world.Best
>>  > > > > open source alternative to Google Dataflow, from the recent talks.
>>  > > > > > Higher freedom to customizations, makes it attractive for
>>  > innovation.
>>  > > > > > 27 continuous monthly meetups.
>>  > > > > > Slack is active.Mailing lists are active.
>>  > > > > > 455 meetup members and counting.40 linkedin group members and
>>  > > counting.
>>  > > > > >
>>  > > > > > All of these, just by a few bunch of us.
>>  > > > > >
>>  > > > > > It is too early for 'retirement' talk, IMHO.
>>  > > > > > Let's focus on, making it to TLP.
>>  > > > > > Taking one task or part of it at a time.
>>  > > > > >
>>  > > > > >
>>  > > > > > Thank you./Sree
>>  > > > > >
>>  > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
>>  > > > > huijun.wu.2010@gmail.com> wrote:
>>  > > > > >
>>  > > > > > The community size and activity look steady rather than
>>  dwindling.
>>  > > The
>>  > > > > > heronstreaming slack is still active. The
>>  > > > > conversations/meetups/discussions
>>  > > > > > keep going well.
>>  > > > > > As for 'retirement' I think that would be premature
>>  > > > > >
>>  > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <wangninggm@gmail.com
>>  >
>>  > > wrote:
>>  > > > > >
>>  > > > > >> Hmm.
>>  > > > > >>
>>  > > > > >> Community isn't very active, but there are still works going on
>>  > > (python,
>>  > > > > >> k8s/helm, etc) and a few users relying on the project. IMO it is
>>  > too
>>  > > > > early
>>  > > > > >> to retire.
>>  > > > > >>
>>  > > > > >>
>>  > > > > >>
>>  > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
>>  josh@joshfischer.io
>>  > >
>>  > > > > wrote:
>>  > > > > >>
>>  > > > > >>> Hi All,
>>  > > > > >>>
>>  > > > > >>> It seems the community is dwindling for Heron. I think it is
>>  time
>>  > > to
>>  > > > > >> start
>>  > > > > >>> a discussion on retiring the podling.
>>  > > > > >>>
>>  > > > > >>> Thoughts?
>>  > > > > >>>
>>  > > > > >>> - Josh
>>  > > > > >>>
>>  > > > > >>
>>  > > > >
>>  > > > >
>>  > > >
>>  > >
>>  > >
>>  >
>>

Re: [DISCUSS] Retire Heron

[Dave Fisher <wa...@apache.org>]

Having a developer community that contributes is the primary requirement for graduating to a TLP.

Making releases is a technical task.

Your contributions to Heron need to be made via PRs which show your merit and build the community.

I am very heartened by this discussion. Make Heron a great community.

Regards,
Dave

Sent from my iPhone

> On Oct 16, 2020, at 7:46 AM, Ron Wilcom <rw...@wilcomweb.net> wrote:
> 
> All,
> We have employed Heron on a large streaming data project for our customer and we have a deep vested interest in its adoption/success.  It would be very premature to retire Heron while it has not yet reached a full Apache release.  Heron has been solid for us and we've been using it for well over a year against the largest of data throughputs - its held up great and is the most versatile/configurable/industrial streaming engine when using the straight/original topology approach -- I'm not a fan of the Streamlets API .. why try to compete there? Heron should go for being the best industrial solution!
> 
> Whatever the hold up for this project is preventing it from moving out from incubation should be prioritized for this developer community.  Many companies are not going to use an "incubated" Apache open source product - but once fully released/available in Apache developers will have an easier time convincing their technical and business management that they can move to Heron and trust that it will have staying power.  Here is a list of some tasks that I feel need to be tackled to provide an upswell of adoption:
> 
> * Get out of "Apache incubation" state to a full release - this has to happen before there is real/broad adoption - are we looking for 'perfect', no, because you'll never get there ... perfection can happen over time after more open source adoption!
> * Capitalize on the history of this project: That it is derived from the Twitter development team that Storm projects can easily convert to Heron and why that is important (better performance, deployment versatility, backpressure features, etc).
> * Where are the original founders of this project (Karthik Ramasamy? now at Splunk?) - why do they no longer support this project and can we get them involved again to help get this over the finish line?  Need more high level advocacy all around to promote Heron.
> * The technical barrier to entry with Heron is high, it took me a lot of grunt work years ago to figure out the proper way to make it work - so it is very important to provide strong/deep documentation, getting started documentation, and out-of-the-box multi-language examples
> * Provide a script/helm based end-to-end deployment that can be run on a developers local machine/VM using well known resource managers like Kubernetes (e.g. minikube example) - obviously this requires pulling in multiple supporting technologies ..... provide expanded examples for large/true clusters.
> * Need a better administrative tool and monitoring ability across/within the topologies - we have many separated topologies working together across the stream - its difficult for a new person on the project to see how these link together or how the data flows - ability to easily track a tuple through - built in ability to send 'canary' tuples to insure throughput
> * Explain the best practices for including Heron in a wholistic streaming architecture such as using Kafka or Pulsar between smaller topologies so the stream has queuing break points during backpressure/restarts - how to use local caches (Redis) or the best approaches for writing out to database end points from bolts, etc
> * Correct the default packing algorithm to the original (maybe this is already fixed?) - there was a release a while back where the default packing algorithm was changed to create a container per bolt/spout which is not a good approach on limited hardware (not everyone has 1000 nodes) - the concepts of this need better explanation/understanding
> * Update the deployment process such that the Heron Client is part of compilation but not required to deploy to Kubernetes - result of the Heron Client should be standard container images that can be pushed into a container repository to be pushed normally like any other image --- this will allow it to be more easily mixed in with standard DevOps (DDS) procedures
> * Allow for on-the-fly (per-environment) configuration settings at the point of build/deploy - currently in the version we use its required to rebuild per environment (I think this may be handled in a newer release but we've been waiting for a final release to occur that gets past incubation?)
> * FUTURE: Provide even further configuration for spouts/bolts ... allow for more dynamic CPU/Memory allocation, special assignment to CPUs/GPUs, simplify VM settings, etc
> * FUTURE: Expand on functionality such as elastic scaling (Dhalion) but provide this within more ubiquitous resource managers like Kubernetes (my understanding is that this only works in Mesos/Aurora?).
> 
> Thanks all - keep it up - let me know if I can help (my time is limited - maybe documentation?)
> 
> Ron
> 
> 
> ------ Original Message ------
> From: "Jim Mantheiy" <jm...@gmail.com>
> To: dev@heron.apache.org
> Cc: "Sree Vaddi" <sr...@yahoo.com>; dev@heron.incubator.apache.org
> Sent: 10/16/2020 8:32:07 AM
> Subject: Re: [DISCUSS] Retire Heron
> 
>> All,
>> 
>> I speak for a few people I work with when I say that heron has a unique
>> place in the streaming/analytic space. Have used storm, kafka streams, and
>> other frameworks. I feel heron is easily the most simplistic, lowest cost
>> to entry framework out there.
>> 
>> Personally, if heron would to expand its k8a capabilities such as
>> horizontal pod autoscaler, health checks on bolts,, better dashboard,
>> perhaps open trace? Then heron would be a one stop shop for highly
>> efficient, scalable, robust streaming solution.
>> 
>> Basically, how can I help?
>> 
>> Thanks
>> 
>> Jim
>> 
>>> On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io> wrote:
>>> 
>>> Hi All,
>>> 
>>> Windham, I agree with everything you said. Most importantly what stood out
>>> to me is the lack of documentation that covers why or how someone would use
>>> Heron.  I agree with Dave, we should try to organize and set some goals for
>>> us to complete within the next few weeks and months.
>>> 
>>> I don't want to see Heron go.  It's the first big open source project I've
>>> worked on and I'd hate to get all teary eyed over a bunch of code that
>>> retired at the Apache Foundation.
>>> 
>>> How would everyone feel if we used this github project
>>> https://github.com/apache/incubator-heron/projects/4 to track some tasks?
>>> This way we could give some visibility to people trying to learn what's
>>> going on with the community?
>>> 
>>> What is left outstanding with our 0.23.0-incubating release?  Let's add
>>> those tasks to the github project above.
>>> 
>>>> On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com> wrote:
>>> 
>>> > Thanks! That's a lot of helpful information!
>>> >
>>> > Agreed that documentation and examples can be better to lower the barrier
>>> > and be more friendly to new users.
>>> >
>>> > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <wi...@stormeye.io>
>>> > wrote:
>>> >
>>> > > I am new to Apache foundation thing and I want to point out a thing
>>> that,
>>> > > when start trying Heron, i got a very big barrier that the
>>> documentation
>>> > is
>>> > > not good enough to do quick start and good understanding the structure
>>> of
>>> > > Heron. I saw a few people asking about how to launch the demo topology
>>> > with
>>> > > facing some technical issues related to python version and
>>> > configurations.
>>> > >
>>> > > For my point of view, we are using Heron as production for log parsing
>>> > > system, and we see great opportunity of increasing the usage of Heron
>>> > with
>>> > > our growth of business. However, recalling my experience when started
>>> > > looking into Heron, the learning curve isn't too high but still much
>>> > higher
>>> > > than other software or systems. I believe the documentation requires
>>> more
>>> > > improvement to let new users to understand more quickly. Furthermore,
>>> > for a
>>> > > business aspect, I believe Heron requires more use case promotion to
>>> > > people. Many don't know what to do with a piece of software and they
>>> > forget
>>> > > about it after some time. Cross-language support (Java/Python/Lua/C++)
>>> is
>>> > > great for people in different field to start using it, but they can't
>>> > find
>>> > > a blog/article/tutorial/youtube to realise what they can do with it. I
>>> am
>>> > > thinking if we can ask companies to share their experience of using
>>> > Heron,
>>> > > and also some personal to share what they can do or their idea would
>>> help
>>> > > the community growth.
>>> > > Sorry for the long words.
>>> > > Windham Wong
>>> > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
>>> > > Co-Founder, Managing Partner of
>>> > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
>>> > > Email // windham.wong@stormeye.io (
>>> > >
>>> >
>>> https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
>>> > > )
>>> > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
>>> > > Fax // +852_3590_2202 (tel:+852_3590_2202)
>>> > >
>>> > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com> wrote:
>>> > > > Thanks Dave!
>>> > > >
>>> > > > IMO our goal is to have an official release, which has been
>>> > challenging.
>>> > > At
>>> > > > the same time, some kubernetes and python works are going on at
>>> least.
>>> > I
>>> > > > remember the issue we found in the latest release candidate was
>>> Python
>>> > 3
>>> > > > related.
>>> > > >
>>> > > >
>>> > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wa...@apache.org>
>>> wrote:
>>> > > > > It would be helpful to have more discussion about what is happening
>>> > on
>>> > > > > this mailing list.
>>> > > > >
>>> > > > > I’m your last active Mentor and I joined only when it seemed like
>>> the
>>> > > > > start of incubation was blocked.
>>> > > > >
>>> > > > > Please show the activity with some visible direction.
>>> > > > >
>>> > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
>>> sree_at_chess@yahoo.com
>>> > > .INVALID>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > Heron will continue to live long.
>>> > > > > > It has it's own place in the stream processing world among other
>>> > > > > competing technologies.The ever increasing data has stretched
>>> > > competitions
>>> > > > > to the limits of breaking.
>>> > > > > >
>>> > > > > > In addition:
>>> > > > > > In production at the creating company and others around the
>>> > > world.Best
>>> > > > > open source alternative to Google Dataflow, from the recent talks.
>>> > > > > > Higher freedom to customizations, makes it attractive for
>>> > innovation.
>>> > > > > > 27 continuous monthly meetups.
>>> > > > > > Slack is active.Mailing lists are active.
>>> > > > > > 455 meetup members and counting.40 linkedin group members and
>>> > > counting.
>>> > > > > >
>>> > > > > > All of these, just by a few bunch of us.
>>> > > > > >
>>> > > > > > It is too early for 'retirement' talk, IMHO.
>>> > > > > > Let's focus on, making it to TLP.
>>> > > > > > Taking one task or part of it at a time.
>>> > > > > >
>>> > > > > >
>>> > > > > > Thank you./Sree
>>> > > > > >
>>> > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
>>> > > > > huijun.wu.2010@gmail.com> wrote:
>>> > > > > >
>>> > > > > > The community size and activity look steady rather than
>>> dwindling.
>>> > > The
>>> > > > > > heronstreaming slack is still active. The
>>> > > > > conversations/meetups/discussions
>>> > > > > > keep going well.
>>> > > > > > As for 'retirement' I think that would be premature
>>> > > > > >
>>> > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <wangninggm@gmail.com
>>> >
>>> > > wrote:
>>> > > > > >
>>> > > > > >> Hmm.
>>> > > > > >>
>>> > > > > >> Community isn't very active, but there are still works going on
>>> > > (python,
>>> > > > > >> k8s/helm, etc) and a few users relying on the project. IMO it is
>>> > too
>>> > > > > early
>>> > > > > >> to retire.
>>> > > > > >>
>>> > > > > >>
>>> > > > > >>
>>> > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
>>> josh@joshfischer.io
>>> > >
>>> > > > > wrote:
>>> > > > > >>
>>> > > > > >>> Hi All,
>>> > > > > >>>
>>> > > > > >>> It seems the community is dwindling for Heron. I think it is
>>> time
>>> > > to
>>> > > > > >> start
>>> > > > > >>> a discussion on retiring the podling.
>>> > > > > >>>
>>> > > > > >>> Thoughts?
>>> > > > > >>>
>>> > > > > >>> - Josh
>>> > > > > >>>
>>> > > > > >>
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> >

Re: Re[2]: [DISCUSS] Retire Heron

[Windham Wong <wi...@stormeye.io>]

Keeping one single platform for all would be better. I am happy to add some TODOs of myself of Heron into the page at any time.

Windham Wong
OSWE, OSCP, GCIA, Specialist in Cybersecurity
Co-Founder, Managing Partner of
Stormeye.io, Hong Kong Managed Security Operation Center Limited
Email // windham.wong@stormeye.io (https://link.getmailspring.com/link/FC791CEA-9E85-46D3-8B02-F031DBBDEBCF@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmFwYWNoZS5vcmc%3D)
Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
Fax // +852_3590_2202 (tel:+852_3590_2202)

On 10月 26 2020, at 7:29 晚上, Josh Fischer <jo...@joshfischer.io> wrote:
> Hi All,
>
> Let's keep track of the tasks that are needed to make the 0.23.0-incubating
> release ready. I think one thing we can improve on is the *current state*
> of work of the project. We can use a github project here
> https://github.com/apache/incubator-heron/projects/4
> or we can set up something in the apache Jira space. I'm not sure what's
> easier for people to use. Any preferences?
>
> Let's keep this moving and get it done.
> - Josh
> On Mon, Oct 19, 2020 at 1:05 AM Windham Wong <wi...@stormeye.io>
> wrote:
>
> > For the community, I believe hosting different talks in different places
> > would work. I know we already got a meetup monthly but we still need to
> > keep on exploring other talks or places to share the thoughts. Use case of
> > Heron is also a good topic as well, if someone is interested in youtube,
> > this could be very good to increase the noise of Heron and letting more
> > people to join in.
> >
> > Windham Wong
> > Managing Partner, Technical Director of
> > Stormeye.io, Hong Kong Managed Security Operation Center Limited
> > Email // windham.wong@stormeye.io (
> > https://link.getmailspring.com/link/91640592-0118-48BD-992E-AD0173C0CB58@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmFwYWNoZS5vcmc%3D
> > )
> > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> > Fax // +852_3590_2202 (tel:+852_3590_2202)
> >
> > On 10月 16 2020, at 11:00 晚上, Josh Fischer <jo...@joshfischer.io> wrote:
> > > I think we need to focus on things we can control. How can we build the
> > > community? What have we all done at our own companies that has eased the
> > > use of Heron that we can contribute back to the project? Open source only
> > > works if we all give back.
> > >
> > > If we want Heron to succeed we need to be the ones to get Heron out of
> > the
> > > incubator. What improvements/deployment tricks have we learned along the
> > > way of using Heron that we can contribute back to the community either by
> > > code or documentation?
> > >
> > > -- Josh
> > > On Fri, Oct 16, 2020 at 9:46 AM Ron Wilcom <rw...@wilcomweb.net>
> > wrote:
> > > > All,
> > > > We have employed Heron on a large streaming data project for our
> > > > customer and we have a deep vested interest in its adoption/success. It
> > > > would be very premature to retire Heron while it has not yet reached a
> > > > full Apache release. Heron has been solid for us and we've been using
> > > > it for well over a year against the largest of data throughputs - its
> > > > held up great and is the most versatile/configurable/industrial
> > > > streaming engine when using the straight/original topology approach --
> > > > I'm not a fan of the Streamlets API .. why try to compete there? Heron
> > > > should go for being the best industrial solution!
> > > >
> > > > Whatever the hold up for this project is preventing it from moving out
> > > > from incubation should be prioritized for this developer community.
> > > > Many companies are not going to use an "incubated" Apache open source
> > > > product - but once fully released/available in Apache developers will
> > > > have an easier time convincing their technical and business management
> > > > that they can move to Heron and trust that it will have staying power.
> > > > Here is a list of some tasks that I feel need to be tackled to provide
> > > > an upswell of adoption:
> > > >
> > > > * Get out of "Apache incubation" state to a full release - this has to
> > > > happen before there is real/broad adoption - are we looking for
> > > > 'perfect', no, because you'll never get there ... perfection can happen
> > > > over time after more open source adoption!
> > > > * Capitalize on the history of this project: That it is derived from
> > the
> > > > Twitter development team that Storm projects can easily convert to
> > Heron
> > > > and why that is important (better performance, deployment versatility,
> > > > backpressure features, etc).
> > > > * Where are the original founders of this project (Karthik Ramasamy?
> > now
> > > > at Splunk?) - why do they no longer support this project and can we get
> > > > them involved again to help get this over the finish line? Need more
> > > > high level advocacy all around to promote Heron.
> > > > * The technical barrier to entry with Heron is high, it took me a lot
> > of
> > > > grunt work years ago to figure out the proper way to make it work - so
> > > > it is very important to provide strong/deep documentation, getting
> > > > started documentation, and out-of-the-box multi-language examples
> > > > * Provide a script/helm based end-to-end deployment that can be run on
> > a
> > > > developers local machine/VM using well known resource managers like
> > > > Kubernetes (e.g. minikube example) - obviously this requires pulling in
> > > > multiple supporting technologies ..... provide expanded examples for
> > > > large/true clusters.
> > > > * Need a better administrative tool and monitoring ability
> > across/within
> > > > the topologies - we have many separated topologies working together
> > > > across the stream - its difficult for a new person on the project to
> > see
> > > > how these link together or how the data flows - ability to easily track
> > > > a tuple through - built in ability to send 'canary' tuples to insure
> > > > throughput
> > > > * Explain the best practices for including Heron in a wholistic
> > > > streaming architecture such as using Kafka or Pulsar between smaller
> > > > topologies so the stream has queuing break points during
> > > > backpressure/restarts - how to use local caches (Redis) or the best
> > > > approaches for writing out to database end points from bolts, etc
> > > > * Correct the default packing algorithm to the original (maybe this is
> > > > already fixed?) - there was a release a while back where the default
> > > > packing algorithm was changed to create a container per bolt/spout
> > which
> > > > is not a good approach on limited hardware (not everyone has 1000
> > nodes)
> > > > - the concepts of this need better explanation/understanding
> > > > * Update the deployment process such that the Heron Client is part of
> > > > compilation but not required to deploy to Kubernetes - result of the
> > > > Heron Client should be standard container images that can be pushed
> > into
> > > > a container repository to be pushed normally like any other image ---
> > > > this will allow it to be more easily mixed in with standard DevOps
> > (DDS)
> > > > procedures
> > > > * Allow for on-the-fly (per-environment) configuration settings at the
> > > > point of build/deploy - currently in the version we use its required to
> > > > rebuild per environment (I think this may be handled in a newer release
> > > > but we've been waiting for a final release to occur that gets past
> > > > incubation?)
> > > > * FUTURE: Provide even further configuration for spouts/bolts ... allow
> > > > for more dynamic CPU/Memory allocation, special assignment to
> > CPUs/GPUs,
> > > > simplify VM settings, etc
> > > > * FUTURE: Expand on functionality such as elastic scaling (Dhalion) but
> > > > provide this within more ubiquitous resource managers like Kubernetes
> > > > (my understanding is that this only works in Mesos/Aurora?).
> > > >
> > > > Thanks all - keep it up - let me know if I can help (my time is limited
> > > > - maybe documentation?)
> > > >
> > > > Ron
> > > >
> > > >
> > > > ------ Original Message ------
> > > > From: "Jim Mantheiy" <jm...@gmail.com>
> > > > To: dev@heron.apache.org
> > > > Cc: "Sree Vaddi" <sr...@yahoo.com>;
> > > > dev@heron.incubator.apache.org
> > > > Sent: 10/16/2020 8:32:07 AM
> > > > Subject: Re: [DISCUSS] Retire Heron
> > > >
> > > > >All,
> > > > >
> > > > >I speak for a few people I work with when I say that heron has a
> > unique
> > > > >place in the streaming/analytic space. Have used storm, kafka
> > streams, and
> > > > >other frameworks. I feel heron is easily the most simplistic, lowest
> > cost
> > > > >to entry framework out there.
> > > > >
> > > > >Personally, if heron would to expand its k8a capabilities such as
> > > > >horizontal pod autoscaler, health checks on bolts,, better dashboard,
> > > > >perhaps open trace? Then heron would be a one stop shop for highly
> > > > >efficient, scalable, robust streaming solution.
> > > > >
> > > > >Basically, how can I help?
> > > > >
> > > > >Thanks
> > > > >
> > > > >Jim
> > > > >
> > > > >On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io>
> > wrote:
> > > > >
> > > > >> Hi All,
> > > > >>
> > > > >> Windham, I agree with everything you said. Most importantly what
> > stood
> > > > out
> > > > >> to me is the lack of documentation that covers why or how someone
> > > > would use
> > > > >> Heron. I agree with Dave, we should try to organize and set some
> > > > goals for
> > > > >> us to complete within the next few weeks and months.
> > > > >>
> > > > >> I don't want to see Heron go. It's the first big open source project
> > > > I've
> > > > >> worked on and I'd hate to get all teary eyed over a bunch of code
> > that
> > > > >> retired at the Apache Foundation.
> > > > >>
> > > > >> How would everyone feel if we used this github project
> > > > >> https://github.com/apache/incubator-heron/projects/4 to track some
> > > > tasks?
> > > > >> This way we could give some visibility to people trying to learn
> > what's
> > > > >> going on with the community?
> > > > >>
> > > > >> What is left outstanding with our 0.23.0-incubating release? Let's
> > add
> > > > >> those tasks to the github project above.
> > > > >>
> > > > >> On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > Thanks! That's a lot of helpful information!
> > > > >> >
> > > > >> > Agreed that documentation and examples can be better to lower the
> > > > barrier
> > > > >> > and be more friendly to new users.
> > > > >> >
> > > > >> > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <
> > > > windham.wong@stormeye.io>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I am new to Apache foundation thing and I want to point out a
> > thing
> > > > >> that,
> > > > >> > > when start trying Heron, i got a very big barrier that the
> > > > >> documentation
> > > > >> > is
> > > > >> > > not good enough to do quick start and good understanding the
> > > > structure
> > > > >> of
> > > > >> > > Heron. I saw a few people asking about how to launch the demo
> > > > topology
> > > > >> > with
> > > > >> > > facing some technical issues related to python version and
> > > > >> > configurations.
> > > > >> > >
> > > > >> > > For my point of view, we are using Heron as production for log
> > > > parsing
> > > > >> > > system, and we see great opportunity of increasing the usage of
> > > > Heron
> > > > >> > with
> > > > >> > > our growth of business. However, recalling my experience when
> > > > started
> > > > >> > > looking into Heron, the learning curve isn't too high but still
> > > > much
> > > > >> > higher
> > > > >> > > than other software or systems. I believe the documentation
> > > > requires
> > > > >> more
> > > > >> > > improvement to let new users to understand more quickly.
> > > > Furthermore,
> > > > >> > for a
> > > > >> > > business aspect, I believe Heron requires more use case
> > promotion
> > > > to
> > > > >> > > people. Many don't know what to do with a piece of software and
> > > > they
> > > > >> > forget
> > > > >> > > about it after some time. Cross-language support
> > > > (Java/Python/Lua/C++)
> > > > >> is
> > > > >> > > great for people in different field to start using it, but they
> > > > can't
> > > > >> > find
> > > > >> > > a blog/article/tutorial/youtube to realise what they can do with
> > > > it. I
> > > > >> am
> > > > >> > > thinking if we can ask companies to share their experience of
> > using
> > > > >> > Heron,
> > > > >> > > and also some personal to share what they can do or their idea
> > > > would
> > > > >> help
> > > > >> > > the community growth.
> > > > >> > > Sorry for the long words.
> > > > >> > > Windham Wong
> > > > >> > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
> > > > >> > > Co-Founder, Managing Partner of
> > > > >> > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
> > > > >> > > Email // windham.wong@stormeye.io (
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
> > > > >> > > )
> > > > >> > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> > > > >> > > Fax // +852_3590_2202 (tel:+852_3590_2202)
> > > > >> > >
> > > > >> > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com>
> > > > wrote:
> > > > >> > > > Thanks Dave!
> > > > >> > > >
> > > > >> > > > IMO our goal is to have an official release, which has been
> > > > >> > challenging.
> > > > >> > > At
> > > > >> > > > the same time, some kubernetes and python works are going on
> > at
> > > > >> least.
> > > > >> > I
> > > > >> > > > remember the issue we found in the latest release candidate
> > was
> > > > >> Python
> > > > >> > 3
> > > > >> > > > related.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wave@apache.org
> > >
> > > > >> wrote:
> > > > >> > > > > It would be helpful to have more discussion about what is
> > > > happening
> > > > >> > on
> > > > >> > > > > this mailing list.
> > > > >> > > > >
> > > > >> > > > > I’m your last active Mentor and I joined only when it seemed
> > > > like
> > > > >> the
> > > > >> > > > > start of incubation was blocked.
> > > > >> > > > >
> > > > >> > > > > Please show the activity with some visible direction.
> > > > >> > > > >
> > > > >> > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
> > > > >> sree_at_chess@yahoo.com
> > > > >> > > .INVALID>
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > Heron will continue to live long.
> > > > >> > > > > > It has it's own place in the stream processing world among
> > > > other
> > > > >> > > > > competing technologies.The ever increasing data has
> > stretched
> > > > >> > > competitions
> > > > >> > > > > to the limits of breaking.
> > > > >> > > > > >
> > > > >> > > > > > In addition:
> > > > >> > > > > > In production at the creating company and others around
> > the
> > > > >> > > world.Best
> > > > >> > > > > open source alternative to Google Dataflow, from the recent
> > > > talks.
> > > > >> > > > > > Higher freedom to customizations, makes it attractive for
> > > > >> > innovation.
> > > > >> > > > > > 27 continuous monthly meetups.
> > > > >> > > > > > Slack is active.Mailing lists are active.
> > > > >> > > > > > 455 meetup members and counting.40 linkedin group members
> > and
> > > > >> > > counting.
> > > > >> > > > > >
> > > > >> > > > > > All of these, just by a few bunch of us.
> > > > >> > > > > >
> > > > >> > > > > > It is too early for 'retirement' talk, IMHO.
> > > > >> > > > > > Let's focus on, making it to TLP.
> > > > >> > > > > > Taking one task or part of it at a time.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Thank you./Sree
> > > > >> > > > > >
> > > > >> > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
> > > > >> > > > > huijun.wu.2010@gmail.com> wrote:
> > > > >> > > > > >
> > > > >> > > > > > The community size and activity look steady rather than
> > > > >> dwindling.
> > > > >> > > The
> > > > >> > > > > > heronstreaming slack is still active. The
> > > > >> > > > > conversations/meetups/discussions
> > > > >> > > > > > keep going well.
> > > > >> > > > > > As for 'retirement' I think that would be premature
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <
> > > > wangninggm@gmail.com
> > > > >> >
> > > > >> > > wrote:
> > > > >> > > > > >
> > > > >> > > > > >> Hmm.
> > > > >> > > > > >>
> > > > >> > > > > >> Community isn't very active, but there are still works
> > > > going on
> > > > >> > > (python,
> > > > >> > > > > >> k8s/helm, etc) and a few users relying on the project.
> > IMO
> > > > it is
> > > > >> > too
> > > > >> > > > > early
> > > > >> > > > > >> to retire.
> > > > >> > > > > >>
> > > > >> > > > > >>
> > > > >> > > > > >>
> > > > >> > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
> > > > >> josh@joshfischer.io
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > >>
> > > > >> > > > > >>> Hi All,
> > > > >> > > > > >>>
> > > > >> > > > > >>> It seems the community is dwindling for Heron. I think
> > it
> > > > is
> > > > >> time
> > > > >> > > to
> > > > >> > > > > >> start
> > > > >> > > > > >>> a discussion on retiring the podling.
> > > > >> > > > > >>>
> > > > >> > > > > >>> Thoughts?
> > > > >> > > > > >>>
> > > > >> > > > > >>> - Josh
> > > > >> > > > > >>>
> > > > >> > > > > >>
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> >
>

Re: Re[2]: [DISCUSS] Retire Heron

[Josh Fischer <jo...@joshfischer.io>]

Hi All,

Let's keep track of the tasks that are needed to make the 0.23.0-incubating
release ready.  I think one thing we can improve on is the *current state*
of work of the project.    We can use a github project here
https://github.com/apache/incubator-heron/projects/4
or we can set up something in the apache Jira space.  I'm not sure what's
easier for people to use.  Any preferences?

Let's keep this moving and get it done.

- Josh

On Mon, Oct 19, 2020 at 1:05 AM Windham Wong <wi...@stormeye.io>
wrote:

> For the community, I believe hosting different talks in different places
> would work. I know we already got a meetup monthly but we still need to
> keep on exploring other talks or places to share the thoughts. Use case of
> Heron is also a good topic as well, if someone is interested in youtube,
> this could be very good to increase the noise of Heron and letting more
> people to join in.
>
> Windham Wong
> Managing Partner, Technical Director of
> Stormeye.io, Hong Kong Managed Security Operation Center Limited
> Email // windham.wong@stormeye.io (
> https://link.getmailspring.com/link/91640592-0118-48BD-992E-AD0173C0CB58@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmFwYWNoZS5vcmc%3D
> )
> Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> Fax // +852_3590_2202 (tel:+852_3590_2202)
>
> On 10月 16 2020, at 11:00 晚上, Josh Fischer <jo...@joshfischer.io> wrote:
> > I think we need to focus on things we can control. How can we build the
> > community? What have we all done at our own companies that has eased the
> > use of Heron that we can contribute back to the project? Open source only
> > works if we all give back.
> >
> > If we want Heron to succeed we need to be the ones to get Heron out of
> the
> > incubator. What improvements/deployment tricks have we learned along the
> > way of using Heron that we can contribute back to the community either by
> > code or documentation?
> >
> > -- Josh
> > On Fri, Oct 16, 2020 at 9:46 AM Ron Wilcom <rw...@wilcomweb.net>
> wrote:
> > > All,
> > > We have employed Heron on a large streaming data project for our
> > > customer and we have a deep vested interest in its adoption/success. It
> > > would be very premature to retire Heron while it has not yet reached a
> > > full Apache release. Heron has been solid for us and we've been using
> > > it for well over a year against the largest of data throughputs - its
> > > held up great and is the most versatile/configurable/industrial
> > > streaming engine when using the straight/original topology approach --
> > > I'm not a fan of the Streamlets API .. why try to compete there? Heron
> > > should go for being the best industrial solution!
> > >
> > > Whatever the hold up for this project is preventing it from moving out
> > > from incubation should be prioritized for this developer community.
> > > Many companies are not going to use an "incubated" Apache open source
> > > product - but once fully released/available in Apache developers will
> > > have an easier time convincing their technical and business management
> > > that they can move to Heron and trust that it will have staying power.
> > > Here is a list of some tasks that I feel need to be tackled to provide
> > > an upswell of adoption:
> > >
> > > * Get out of "Apache incubation" state to a full release - this has to
> > > happen before there is real/broad adoption - are we looking for
> > > 'perfect', no, because you'll never get there ... perfection can happen
> > > over time after more open source adoption!
> > > * Capitalize on the history of this project: That it is derived from
> the
> > > Twitter development team that Storm projects can easily convert to
> Heron
> > > and why that is important (better performance, deployment versatility,
> > > backpressure features, etc).
> > > * Where are the original founders of this project (Karthik Ramasamy?
> now
> > > at Splunk?) - why do they no longer support this project and can we get
> > > them involved again to help get this over the finish line? Need more
> > > high level advocacy all around to promote Heron.
> > > * The technical barrier to entry with Heron is high, it took me a lot
> of
> > > grunt work years ago to figure out the proper way to make it work - so
> > > it is very important to provide strong/deep documentation, getting
> > > started documentation, and out-of-the-box multi-language examples
> > > * Provide a script/helm based end-to-end deployment that can be run on
> a
> > > developers local machine/VM using well known resource managers like
> > > Kubernetes (e.g. minikube example) - obviously this requires pulling in
> > > multiple supporting technologies ..... provide expanded examples for
> > > large/true clusters.
> > > * Need a better administrative tool and monitoring ability
> across/within
> > > the topologies - we have many separated topologies working together
> > > across the stream - its difficult for a new person on the project to
> see
> > > how these link together or how the data flows - ability to easily track
> > > a tuple through - built in ability to send 'canary' tuples to insure
> > > throughput
> > > * Explain the best practices for including Heron in a wholistic
> > > streaming architecture such as using Kafka or Pulsar between smaller
> > > topologies so the stream has queuing break points during
> > > backpressure/restarts - how to use local caches (Redis) or the best
> > > approaches for writing out to database end points from bolts, etc
> > > * Correct the default packing algorithm to the original (maybe this is
> > > already fixed?) - there was a release a while back where the default
> > > packing algorithm was changed to create a container per bolt/spout
> which
> > > is not a good approach on limited hardware (not everyone has 1000
> nodes)
> > > - the concepts of this need better explanation/understanding
> > > * Update the deployment process such that the Heron Client is part of
> > > compilation but not required to deploy to Kubernetes - result of the
> > > Heron Client should be standard container images that can be pushed
> into
> > > a container repository to be pushed normally like any other image ---
> > > this will allow it to be more easily mixed in with standard DevOps
> (DDS)
> > > procedures
> > > * Allow for on-the-fly (per-environment) configuration settings at the
> > > point of build/deploy - currently in the version we use its required to
> > > rebuild per environment (I think this may be handled in a newer release
> > > but we've been waiting for a final release to occur that gets past
> > > incubation?)
> > > * FUTURE: Provide even further configuration for spouts/bolts ... allow
> > > for more dynamic CPU/Memory allocation, special assignment to
> CPUs/GPUs,
> > > simplify VM settings, etc
> > > * FUTURE: Expand on functionality such as elastic scaling (Dhalion) but
> > > provide this within more ubiquitous resource managers like Kubernetes
> > > (my understanding is that this only works in Mesos/Aurora?).
> > >
> > > Thanks all - keep it up - let me know if I can help (my time is limited
> > > - maybe documentation?)
> > >
> > > Ron
> > >
> > >
> > > ------ Original Message ------
> > > From: "Jim Mantheiy" <jm...@gmail.com>
> > > To: dev@heron.apache.org
> > > Cc: "Sree Vaddi" <sr...@yahoo.com>;
> > > dev@heron.incubator.apache.org
> > > Sent: 10/16/2020 8:32:07 AM
> > > Subject: Re: [DISCUSS] Retire Heron
> > >
> > > >All,
> > > >
> > > >I speak for a few people I work with when I say that heron has a
> unique
> > > >place in the streaming/analytic space. Have used storm, kafka
> streams, and
> > > >other frameworks. I feel heron is easily the most simplistic, lowest
> cost
> > > >to entry framework out there.
> > > >
> > > >Personally, if heron would to expand its k8a capabilities such as
> > > >horizontal pod autoscaler, health checks on bolts,, better dashboard,
> > > >perhaps open trace? Then heron would be a one stop shop for highly
> > > >efficient, scalable, robust streaming solution.
> > > >
> > > >Basically, how can I help?
> > > >
> > > >Thanks
> > > >
> > > >Jim
> > > >
> > > >On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io>
> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> Windham, I agree with everything you said. Most importantly what
> stood
> > > out
> > > >> to me is the lack of documentation that covers why or how someone
> > > would use
> > > >> Heron. I agree with Dave, we should try to organize and set some
> > > goals for
> > > >> us to complete within the next few weeks and months.
> > > >>
> > > >> I don't want to see Heron go. It's the first big open source project
> > > I've
> > > >> worked on and I'd hate to get all teary eyed over a bunch of code
> that
> > > >> retired at the Apache Foundation.
> > > >>
> > > >> How would everyone feel if we used this github project
> > > >> https://github.com/apache/incubator-heron/projects/4 to track some
> > > tasks?
> > > >> This way we could give some visibility to people trying to learn
> what's
> > > >> going on with the community?
> > > >>
> > > >> What is left outstanding with our 0.23.0-incubating release? Let's
> add
> > > >> those tasks to the github project above.
> > > >>
> > > >> On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Thanks! That's a lot of helpful information!
> > > >> >
> > > >> > Agreed that documentation and examples can be better to lower the
> > > barrier
> > > >> > and be more friendly to new users.
> > > >> >
> > > >> > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <
> > > windham.wong@stormeye.io>
> > > >> > wrote:
> > > >> >
> > > >> > > I am new to Apache foundation thing and I want to point out a
> thing
> > > >> that,
> > > >> > > when start trying Heron, i got a very big barrier that the
> > > >> documentation
> > > >> > is
> > > >> > > not good enough to do quick start and good understanding the
> > > structure
> > > >> of
> > > >> > > Heron. I saw a few people asking about how to launch the demo
> > > topology
> > > >> > with
> > > >> > > facing some technical issues related to python version and
> > > >> > configurations.
> > > >> > >
> > > >> > > For my point of view, we are using Heron as production for log
> > > parsing
> > > >> > > system, and we see great opportunity of increasing the usage of
> > > Heron
> > > >> > with
> > > >> > > our growth of business. However, recalling my experience when
> > > started
> > > >> > > looking into Heron, the learning curve isn't too high but still
> > > much
> > > >> > higher
> > > >> > > than other software or systems. I believe the documentation
> > > requires
> > > >> more
> > > >> > > improvement to let new users to understand more quickly.
> > > Furthermore,
> > > >> > for a
> > > >> > > business aspect, I believe Heron requires more use case
> promotion
> > > to
> > > >> > > people. Many don't know what to do with a piece of software and
> > > they
> > > >> > forget
> > > >> > > about it after some time. Cross-language support
> > > (Java/Python/Lua/C++)
> > > >> is
> > > >> > > great for people in different field to start using it, but they
> > > can't
> > > >> > find
> > > >> > > a blog/article/tutorial/youtube to realise what they can do with
> > > it. I
> > > >> am
> > > >> > > thinking if we can ask companies to share their experience of
> using
> > > >> > Heron,
> > > >> > > and also some personal to share what they can do or their idea
> > > would
> > > >> help
> > > >> > > the community growth.
> > > >> > > Sorry for the long words.
> > > >> > > Windham Wong
> > > >> > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
> > > >> > > Co-Founder, Managing Partner of
> > > >> > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
> > > >> > > Email // windham.wong@stormeye.io (
> > > >> > >
> > > >> >
> > > >>
> > >
> https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
> > > >> > > )
> > > >> > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> > > >> > > Fax // +852_3590_2202 (tel:+852_3590_2202)
> > > >> > >
> > > >> > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com>
> > > wrote:
> > > >> > > > Thanks Dave!
> > > >> > > >
> > > >> > > > IMO our goal is to have an official release, which has been
> > > >> > challenging.
> > > >> > > At
> > > >> > > > the same time, some kubernetes and python works are going on
> at
> > > >> least.
> > > >> > I
> > > >> > > > remember the issue we found in the latest release candidate
> was
> > > >> Python
> > > >> > 3
> > > >> > > > related.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wave@apache.org
> >
> > > >> wrote:
> > > >> > > > > It would be helpful to have more discussion about what is
> > > happening
> > > >> > on
> > > >> > > > > this mailing list.
> > > >> > > > >
> > > >> > > > > I’m your last active Mentor and I joined only when it seemed
> > > like
> > > >> the
> > > >> > > > > start of incubation was blocked.
> > > >> > > > >
> > > >> > > > > Please show the activity with some visible direction.
> > > >> > > > >
> > > >> > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
> > > >> sree_at_chess@yahoo.com
> > > >> > > .INVALID>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > Heron will continue to live long.
> > > >> > > > > > It has it's own place in the stream processing world among
> > > other
> > > >> > > > > competing technologies.The ever increasing data has
> stretched
> > > >> > > competitions
> > > >> > > > > to the limits of breaking.
> > > >> > > > > >
> > > >> > > > > > In addition:
> > > >> > > > > > In production at the creating company and others around
> the
> > > >> > > world.Best
> > > >> > > > > open source alternative to Google Dataflow, from the recent
> > > talks.
> > > >> > > > > > Higher freedom to customizations, makes it attractive for
> > > >> > innovation.
> > > >> > > > > > 27 continuous monthly meetups.
> > > >> > > > > > Slack is active.Mailing lists are active.
> > > >> > > > > > 455 meetup members and counting.40 linkedin group members
> and
> > > >> > > counting.
> > > >> > > > > >
> > > >> > > > > > All of these, just by a few bunch of us.
> > > >> > > > > >
> > > >> > > > > > It is too early for 'retirement' talk, IMHO.
> > > >> > > > > > Let's focus on, making it to TLP.
> > > >> > > > > > Taking one task or part of it at a time.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Thank you./Sree
> > > >> > > > > >
> > > >> > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
> > > >> > > > > huijun.wu.2010@gmail.com> wrote:
> > > >> > > > > >
> > > >> > > > > > The community size and activity look steady rather than
> > > >> dwindling.
> > > >> > > The
> > > >> > > > > > heronstreaming slack is still active. The
> > > >> > > > > conversations/meetups/discussions
> > > >> > > > > > keep going well.
> > > >> > > > > > As for 'retirement' I think that would be premature
> > > >> > > > > >
> > > >> > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <
> > > wangninggm@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > > >
> > > >> > > > > >> Hmm.
> > > >> > > > > >>
> > > >> > > > > >> Community isn't very active, but there are still works
> > > going on
> > > >> > > (python,
> > > >> > > > > >> k8s/helm, etc) and a few users relying on the project.
> IMO
> > > it is
> > > >> > too
> > > >> > > > > early
> > > >> > > > > >> to retire.
> > > >> > > > > >>
> > > >> > > > > >>
> > > >> > > > > >>
> > > >> > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
> > > >> josh@joshfischer.io
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > >>
> > > >> > > > > >>> Hi All,
> > > >> > > > > >>>
> > > >> > > > > >>> It seems the community is dwindling for Heron. I think
> it
> > > is
> > > >> time
> > > >> > > to
> > > >> > > > > >> start
> > > >> > > > > >>> a discussion on retiring the podling.
> > > >> > > > > >>>
> > > >> > > > > >>> Thoughts?
> > > >> > > > > >>>
> > > >> > > > > >>> - Josh
> > > >> > > > > >>>
> > > >> > > > > >>
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> >
>
>

Re: Re[2]: [DISCUSS] Retire Heron

[Windham Wong <wi...@stormeye.io>]

For the community, I believe hosting different talks in different places would work. I know we already got a meetup monthly but we still need to keep on exploring other talks or places to share the thoughts. Use case of Heron is also a good topic as well, if someone is interested in youtube, this could be very good to increase the noise of Heron and letting more people to join in.

Windham Wong
Managing Partner, Technical Director of
Stormeye.io, Hong Kong Managed Security Operation Center Limited
Email // windham.wong@stormeye.io (https://link.getmailspring.com/link/91640592-0118-48BD-992E-AD0173C0CB58@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmFwYWNoZS5vcmc%3D)
Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
Fax // +852_3590_2202 (tel:+852_3590_2202)

On 10月 16 2020, at 11:00 晚上, Josh Fischer <jo...@joshfischer.io> wrote:
> I think we need to focus on things we can control. How can we build the
> community? What have we all done at our own companies that has eased the
> use of Heron that we can contribute back to the project? Open source only
> works if we all give back.
>
> If we want Heron to succeed we need to be the ones to get Heron out of the
> incubator. What improvements/deployment tricks have we learned along the
> way of using Heron that we can contribute back to the community either by
> code or documentation?
>
> -- Josh
> On Fri, Oct 16, 2020 at 9:46 AM Ron Wilcom <rw...@wilcomweb.net> wrote:
> > All,
> > We have employed Heron on a large streaming data project for our
> > customer and we have a deep vested interest in its adoption/success. It
> > would be very premature to retire Heron while it has not yet reached a
> > full Apache release. Heron has been solid for us and we've been using
> > it for well over a year against the largest of data throughputs - its
> > held up great and is the most versatile/configurable/industrial
> > streaming engine when using the straight/original topology approach --
> > I'm not a fan of the Streamlets API .. why try to compete there? Heron
> > should go for being the best industrial solution!
> >
> > Whatever the hold up for this project is preventing it from moving out
> > from incubation should be prioritized for this developer community.
> > Many companies are not going to use an "incubated" Apache open source
> > product - but once fully released/available in Apache developers will
> > have an easier time convincing their technical and business management
> > that they can move to Heron and trust that it will have staying power.
> > Here is a list of some tasks that I feel need to be tackled to provide
> > an upswell of adoption:
> >
> > * Get out of "Apache incubation" state to a full release - this has to
> > happen before there is real/broad adoption - are we looking for
> > 'perfect', no, because you'll never get there ... perfection can happen
> > over time after more open source adoption!
> > * Capitalize on the history of this project: That it is derived from the
> > Twitter development team that Storm projects can easily convert to Heron
> > and why that is important (better performance, deployment versatility,
> > backpressure features, etc).
> > * Where are the original founders of this project (Karthik Ramasamy? now
> > at Splunk?) - why do they no longer support this project and can we get
> > them involved again to help get this over the finish line? Need more
> > high level advocacy all around to promote Heron.
> > * The technical barrier to entry with Heron is high, it took me a lot of
> > grunt work years ago to figure out the proper way to make it work - so
> > it is very important to provide strong/deep documentation, getting
> > started documentation, and out-of-the-box multi-language examples
> > * Provide a script/helm based end-to-end deployment that can be run on a
> > developers local machine/VM using well known resource managers like
> > Kubernetes (e.g. minikube example) - obviously this requires pulling in
> > multiple supporting technologies ..... provide expanded examples for
> > large/true clusters.
> > * Need a better administrative tool and monitoring ability across/within
> > the topologies - we have many separated topologies working together
> > across the stream - its difficult for a new person on the project to see
> > how these link together or how the data flows - ability to easily track
> > a tuple through - built in ability to send 'canary' tuples to insure
> > throughput
> > * Explain the best practices for including Heron in a wholistic
> > streaming architecture such as using Kafka or Pulsar between smaller
> > topologies so the stream has queuing break points during
> > backpressure/restarts - how to use local caches (Redis) or the best
> > approaches for writing out to database end points from bolts, etc
> > * Correct the default packing algorithm to the original (maybe this is
> > already fixed?) - there was a release a while back where the default
> > packing algorithm was changed to create a container per bolt/spout which
> > is not a good approach on limited hardware (not everyone has 1000 nodes)
> > - the concepts of this need better explanation/understanding
> > * Update the deployment process such that the Heron Client is part of
> > compilation but not required to deploy to Kubernetes - result of the
> > Heron Client should be standard container images that can be pushed into
> > a container repository to be pushed normally like any other image ---
> > this will allow it to be more easily mixed in with standard DevOps (DDS)
> > procedures
> > * Allow for on-the-fly (per-environment) configuration settings at the
> > point of build/deploy - currently in the version we use its required to
> > rebuild per environment (I think this may be handled in a newer release
> > but we've been waiting for a final release to occur that gets past
> > incubation?)
> > * FUTURE: Provide even further configuration for spouts/bolts ... allow
> > for more dynamic CPU/Memory allocation, special assignment to CPUs/GPUs,
> > simplify VM settings, etc
> > * FUTURE: Expand on functionality such as elastic scaling (Dhalion) but
> > provide this within more ubiquitous resource managers like Kubernetes
> > (my understanding is that this only works in Mesos/Aurora?).
> >
> > Thanks all - keep it up - let me know if I can help (my time is limited
> > - maybe documentation?)
> >
> > Ron
> >
> >
> > ------ Original Message ------
> > From: "Jim Mantheiy" <jm...@gmail.com>
> > To: dev@heron.apache.org
> > Cc: "Sree Vaddi" <sr...@yahoo.com>;
> > dev@heron.incubator.apache.org
> > Sent: 10/16/2020 8:32:07 AM
> > Subject: Re: [DISCUSS] Retire Heron
> >
> > >All,
> > >
> > >I speak for a few people I work with when I say that heron has a unique
> > >place in the streaming/analytic space. Have used storm, kafka streams, and
> > >other frameworks. I feel heron is easily the most simplistic, lowest cost
> > >to entry framework out there.
> > >
> > >Personally, if heron would to expand its k8a capabilities such as
> > >horizontal pod autoscaler, health checks on bolts,, better dashboard,
> > >perhaps open trace? Then heron would be a one stop shop for highly
> > >efficient, scalable, robust streaming solution.
> > >
> > >Basically, how can I help?
> > >
> > >Thanks
> > >
> > >Jim
> > >
> > >On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io> wrote:
> > >
> > >> Hi All,
> > >>
> > >> Windham, I agree with everything you said. Most importantly what stood
> > out
> > >> to me is the lack of documentation that covers why or how someone
> > would use
> > >> Heron. I agree with Dave, we should try to organize and set some
> > goals for
> > >> us to complete within the next few weeks and months.
> > >>
> > >> I don't want to see Heron go. It's the first big open source project
> > I've
> > >> worked on and I'd hate to get all teary eyed over a bunch of code that
> > >> retired at the Apache Foundation.
> > >>
> > >> How would everyone feel if we used this github project
> > >> https://github.com/apache/incubator-heron/projects/4 to track some
> > tasks?
> > >> This way we could give some visibility to people trying to learn what's
> > >> going on with the community?
> > >>
> > >> What is left outstanding with our 0.23.0-incubating release? Let's add
> > >> those tasks to the github project above.
> > >>
> > >> On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com>
> > wrote:
> > >>
> > >> > Thanks! That's a lot of helpful information!
> > >> >
> > >> > Agreed that documentation and examples can be better to lower the
> > barrier
> > >> > and be more friendly to new users.
> > >> >
> > >> > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <
> > windham.wong@stormeye.io>
> > >> > wrote:
> > >> >
> > >> > > I am new to Apache foundation thing and I want to point out a thing
> > >> that,
> > >> > > when start trying Heron, i got a very big barrier that the
> > >> documentation
> > >> > is
> > >> > > not good enough to do quick start and good understanding the
> > structure
> > >> of
> > >> > > Heron. I saw a few people asking about how to launch the demo
> > topology
> > >> > with
> > >> > > facing some technical issues related to python version and
> > >> > configurations.
> > >> > >
> > >> > > For my point of view, we are using Heron as production for log
> > parsing
> > >> > > system, and we see great opportunity of increasing the usage of
> > Heron
> > >> > with
> > >> > > our growth of business. However, recalling my experience when
> > started
> > >> > > looking into Heron, the learning curve isn't too high but still
> > much
> > >> > higher
> > >> > > than other software or systems. I believe the documentation
> > requires
> > >> more
> > >> > > improvement to let new users to understand more quickly.
> > Furthermore,
> > >> > for a
> > >> > > business aspect, I believe Heron requires more use case promotion
> > to
> > >> > > people. Many don't know what to do with a piece of software and
> > they
> > >> > forget
> > >> > > about it after some time. Cross-language support
> > (Java/Python/Lua/C++)
> > >> is
> > >> > > great for people in different field to start using it, but they
> > can't
> > >> > find
> > >> > > a blog/article/tutorial/youtube to realise what they can do with
> > it. I
> > >> am
> > >> > > thinking if we can ask companies to share their experience of using
> > >> > Heron,
> > >> > > and also some personal to share what they can do or their idea
> > would
> > >> help
> > >> > > the community growth.
> > >> > > Sorry for the long words.
> > >> > > Windham Wong
> > >> > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
> > >> > > Co-Founder, Managing Partner of
> > >> > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
> > >> > > Email // windham.wong@stormeye.io (
> > >> > >
> > >> >
> > >>
> > https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
> > >> > > )
> > >> > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> > >> > > Fax // +852_3590_2202 (tel:+852_3590_2202)
> > >> > >
> > >> > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com>
> > wrote:
> > >> > > > Thanks Dave!
> > >> > > >
> > >> > > > IMO our goal is to have an official release, which has been
> > >> > challenging.
> > >> > > At
> > >> > > > the same time, some kubernetes and python works are going on at
> > >> least.
> > >> > I
> > >> > > > remember the issue we found in the latest release candidate was
> > >> Python
> > >> > 3
> > >> > > > related.
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wa...@apache.org>
> > >> wrote:
> > >> > > > > It would be helpful to have more discussion about what is
> > happening
> > >> > on
> > >> > > > > this mailing list.
> > >> > > > >
> > >> > > > > I’m your last active Mentor and I joined only when it seemed
> > like
> > >> the
> > >> > > > > start of incubation was blocked.
> > >> > > > >
> > >> > > > > Please show the activity with some visible direction.
> > >> > > > >
> > >> > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
> > >> sree_at_chess@yahoo.com
> > >> > > .INVALID>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > Heron will continue to live long.
> > >> > > > > > It has it's own place in the stream processing world among
> > other
> > >> > > > > competing technologies.The ever increasing data has stretched
> > >> > > competitions
> > >> > > > > to the limits of breaking.
> > >> > > > > >
> > >> > > > > > In addition:
> > >> > > > > > In production at the creating company and others around the
> > >> > > world.Best
> > >> > > > > open source alternative to Google Dataflow, from the recent
> > talks.
> > >> > > > > > Higher freedom to customizations, makes it attractive for
> > >> > innovation.
> > >> > > > > > 27 continuous monthly meetups.
> > >> > > > > > Slack is active.Mailing lists are active.
> > >> > > > > > 455 meetup members and counting.40 linkedin group members and
> > >> > > counting.
> > >> > > > > >
> > >> > > > > > All of these, just by a few bunch of us.
> > >> > > > > >
> > >> > > > > > It is too early for 'retirement' talk, IMHO.
> > >> > > > > > Let's focus on, making it to TLP.
> > >> > > > > > Taking one task or part of it at a time.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thank you./Sree
> > >> > > > > >
> > >> > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
> > >> > > > > huijun.wu.2010@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > The community size and activity look steady rather than
> > >> dwindling.
> > >> > > The
> > >> > > > > > heronstreaming slack is still active. The
> > >> > > > > conversations/meetups/discussions
> > >> > > > > > keep going well.
> > >> > > > > > As for 'retirement' I think that would be premature
> > >> > > > > >
> > >> > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <
> > wangninggm@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > >> Hmm.
> > >> > > > > >>
> > >> > > > > >> Community isn't very active, but there are still works
> > going on
> > >> > > (python,
> > >> > > > > >> k8s/helm, etc) and a few users relying on the project. IMO
> > it is
> > >> > too
> > >> > > > > early
> > >> > > > > >> to retire.
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
> > >> josh@joshfischer.io
> > >> > >
> > >> > > > > wrote:
> > >> > > > > >>
> > >> > > > > >>> Hi All,
> > >> > > > > >>>
> > >> > > > > >>> It seems the community is dwindling for Heron. I think it
> > is
> > >> time
> > >> > > to
> > >> > > > > >> start
> > >> > > > > >>> a discussion on retiring the podling.
> > >> > > > > >>>
> > >> > > > > >>> Thoughts?
> > >> > > > > >>>
> > >> > > > > >>> - Josh
> > >> > > > > >>>
> > >> > > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >>
>

Re: Re[2]: [DISCUSS] Retire Heron

[Josh Fischer <jo...@joshfischer.io>]

I think we need to focus on things we can control.  How can we build the
community?  What have we all done at our own companies that has eased the
use of Heron that we can contribute back to the project?  Open source only
works if we all give back.

If we want Heron to succeed we need to be the ones to get Heron out of the
incubator.   What improvements/deployment tricks have we learned along the
way of using Heron that we can contribute back to the community either by
code or documentation?

-- Josh

On Fri, Oct 16, 2020 at 9:46 AM Ron Wilcom <rw...@wilcomweb.net> wrote:

> All,
> We have employed Heron on a large streaming data project for our
> customer and we have a deep vested interest in its adoption/success.  It
> would be very premature to retire Heron while it has not yet reached a
> full Apache release.  Heron has been solid for us and we've been using
> it for well over a year against the largest of data throughputs - its
> held up great and is the most versatile/configurable/industrial
> streaming engine when using the straight/original topology approach --
> I'm not a fan of the Streamlets API .. why try to compete there? Heron
> should go for being the best industrial solution!
>
> Whatever the hold up for this project is preventing it from moving out
> from incubation should be prioritized for this developer community.
> Many companies are not going to use an "incubated" Apache open source
> product - but once fully released/available in Apache developers will
> have an easier time convincing their technical and business management
> that they can move to Heron and trust that it will have staying power.
> Here is a list of some tasks that I feel need to be tackled to provide
> an upswell of adoption:
>
> * Get out of "Apache incubation" state to a full release - this has to
> happen before there is real/broad adoption - are we looking for
> 'perfect', no, because you'll never get there ... perfection can happen
> over time after more open source adoption!
> * Capitalize on the history of this project: That it is derived from the
> Twitter development team that Storm projects can easily convert to Heron
> and why that is important (better performance, deployment versatility,
> backpressure features, etc).
> * Where are the original founders of this project (Karthik Ramasamy? now
> at Splunk?) - why do they no longer support this project and can we get
> them involved again to help get this over the finish line?  Need more
> high level advocacy all around to promote Heron.
> * The technical barrier to entry with Heron is high, it took me a lot of
> grunt work years ago to figure out the proper way to make it work - so
> it is very important to provide strong/deep documentation, getting
> started documentation, and out-of-the-box multi-language examples
> * Provide a script/helm based end-to-end deployment that can be run on a
> developers local machine/VM using well known resource managers like
> Kubernetes (e.g. minikube example) - obviously this requires pulling in
> multiple supporting technologies ..... provide expanded examples for
> large/true clusters.
> * Need a better administrative tool and monitoring ability across/within
> the topologies - we have many separated topologies working together
> across the stream - its difficult for a new person on the project to see
> how these link together or how the data flows - ability to easily track
> a tuple through - built in ability to send 'canary' tuples to insure
> throughput
> * Explain the best practices for including Heron in a wholistic
> streaming architecture such as using Kafka or Pulsar between smaller
> topologies so the stream has queuing break points during
> backpressure/restarts - how to use local caches (Redis) or the best
> approaches for writing out to database end points from bolts, etc
> * Correct the default packing algorithm to the original (maybe this is
> already fixed?) - there was a release a while back where the default
> packing algorithm was changed to create a container per bolt/spout which
> is not a good approach on limited hardware (not everyone has 1000 nodes)
> - the concepts of this need better explanation/understanding
> * Update the deployment process such that the Heron Client is part of
> compilation but not required to deploy to Kubernetes - result of the
> Heron Client should be standard container images that can be pushed into
> a container repository to be pushed normally like any other image ---
> this will allow it to be more easily mixed in with standard DevOps (DDS)
> procedures
> * Allow for on-the-fly (per-environment) configuration settings at the
> point of build/deploy - currently in the version we use its required to
> rebuild per environment (I think this may be handled in a newer release
> but we've been waiting for a final release to occur that gets past
> incubation?)
> * FUTURE: Provide even further configuration for spouts/bolts ... allow
> for more dynamic CPU/Memory allocation, special assignment to CPUs/GPUs,
> simplify VM settings, etc
> * FUTURE: Expand on functionality such as elastic scaling (Dhalion) but
> provide this within more ubiquitous resource managers like Kubernetes
> (my understanding is that this only works in Mesos/Aurora?).
>
> Thanks all - keep it up - let me know if I can help (my time is limited
> - maybe documentation?)
>
> Ron
>
>
> ------ Original Message ------
> From: "Jim Mantheiy" <jm...@gmail.com>
> To: dev@heron.apache.org
> Cc: "Sree Vaddi" <sr...@yahoo.com>;
> dev@heron.incubator.apache.org
> Sent: 10/16/2020 8:32:07 AM
> Subject: Re: [DISCUSS] Retire Heron
>
> >All,
> >
> >I speak for a few people I work with when I say that heron has a unique
> >place in the streaming/analytic space. Have used storm, kafka streams, and
> >other frameworks. I feel heron is easily the most simplistic, lowest cost
> >to entry framework out there.
> >
> >Personally, if heron would to expand its k8a capabilities such as
> >horizontal pod autoscaler, health checks on bolts,, better dashboard,
> >perhaps open trace? Then heron would be a one stop shop for highly
> >efficient, scalable, robust streaming solution.
> >
> >Basically, how can I help?
> >
> >Thanks
> >
> >Jim
> >
> >On Fri, Oct 16, 2020, 7:33 AM Josh Fischer <jo...@joshfischer.io> wrote:
> >
> >>  Hi All,
> >>
> >>  Windham, I agree with everything you said. Most importantly what stood
> out
> >>  to me is the lack of documentation that covers why or how someone
> would use
> >>  Heron.  I agree with Dave, we should try to organize and set some
> goals for
> >>  us to complete within the next few weeks and months.
> >>
> >>  I don't want to see Heron go.  It's the first big open source project
> I've
> >>  worked on and I'd hate to get all teary eyed over a bunch of code that
> >>  retired at the Apache Foundation.
> >>
> >>  How would everyone feel if we used this github project
> >>  https://github.com/apache/incubator-heron/projects/4 to track some
> tasks?
> >>  This way we could give some visibility to people trying to learn what's
> >>  going on with the community?
> >>
> >>  What is left outstanding with our 0.23.0-incubating release?  Let's add
> >>  those tasks to the github project above.
> >>
> >>  On Thu, Oct 15, 2020 at 9:59 PM Ning Wang <wa...@gmail.com>
> wrote:
> >>
> >>  > Thanks! That's a lot of helpful information!
> >>  >
> >>  > Agreed that documentation and examples can be better to lower the
> barrier
> >>  > and be more friendly to new users.
> >>  >
> >>  > On Thu, Oct 15, 2020 at 7:42 PM Windham Wong <
> windham.wong@stormeye.io>
> >>  > wrote:
> >>  >
> >>  > > I am new to Apache foundation thing and I want to point out a thing
> >>  that,
> >>  > > when start trying Heron, i got a very big barrier that the
> >>  documentation
> >>  > is
> >>  > > not good enough to do quick start and good understanding the
> structure
> >>  of
> >>  > > Heron. I saw a few people asking about how to launch the demo
> topology
> >>  > with
> >>  > > facing some technical issues related to python version and
> >>  > configurations.
> >>  > >
> >>  > > For my point of view, we are using Heron as production for log
> parsing
> >>  > > system, and we see great opportunity of increasing the usage of
> Heron
> >>  > with
> >>  > > our growth of business. However, recalling my experience when
> started
> >>  > > looking into Heron, the learning curve isn't too high but still
> much
> >>  > higher
> >>  > > than other software or systems. I believe the documentation
> requires
> >>  more
> >>  > > improvement to let new users to understand more quickly.
> Furthermore,
> >>  > for a
> >>  > > business aspect, I believe Heron requires more use case promotion
> to
> >>  > > people. Many don't know what to do with a piece of software and
> they
> >>  > forget
> >>  > > about it after some time. Cross-language support
> (Java/Python/Lua/C++)
> >>  is
> >>  > > great for people in different field to start using it, but they
> can't
> >>  > find
> >>  > > a blog/article/tutorial/youtube to realise what they can do with
> it. I
> >>  am
> >>  > > thinking if we can ask companies to share their experience of using
> >>  > Heron,
> >>  > > and also some personal to share what they can do or their idea
> would
> >>  help
> >>  > > the community growth.
> >>  > > Sorry for the long words.
> >>  > > Windham Wong
> >>  > > OSWE, OSCP, GCIA, Specialist in Cybersecurity
> >>  > > Co-Founder, Managing Partner of
> >>  > > Stormeye.io, Hong Kong Managed Security Operation Center Limited
> >>  > > Email // windham.wong@stormeye.io (
> >>  > >
> >>  >
> >>
> https://link.getmailspring.com/link/C4371490-7F73-4F27-888E-F5B8762709A6@getmailspring.com/1?redirect=mailto%3Awindham.wong%40stormeye.io&recipient=ZGV2QGhlcm9uLmluY3ViYXRvci5hcGFjaGUub3Jn
> >>  > > )
> >>  > > Phone // +852_3590_2212_|_+852_9832_0707 (tel:+85235902212)
> >>  > > Fax // +852_3590_2202 (tel:+852_3590_2202)
> >>  > >
> >>  > > On 10月 16 2020, at 4:44 凌晨, Ning Wang <wa...@gmail.com>
> wrote:
> >>  > > > Thanks Dave!
> >>  > > >
> >>  > > > IMO our goal is to have an official release, which has been
> >>  > challenging.
> >>  > > At
> >>  > > > the same time, some kubernetes and python works are going on at
> >>  least.
> >>  > I
> >>  > > > remember the issue we found in the latest release candidate was
> >>  Python
> >>  > 3
> >>  > > > related.
> >>  > > >
> >>  > > >
> >>  > > > On Thu, Oct 15, 2020 at 12:51 PM Dave Fisher <wa...@apache.org>
> >>  wrote:
> >>  > > > > It would be helpful to have more discussion about what is
> happening
> >>  > on
> >>  > > > > this mailing list.
> >>  > > > >
> >>  > > > > I’m your last active Mentor and I joined only when it seemed
> like
> >>  the
> >>  > > > > start of incubation was blocked.
> >>  > > > >
> >>  > > > > Please show the activity with some visible direction.
> >>  > > > >
> >>  > > > > > On Oct 15, 2020, at 11:59 AM, Sree Vaddi <
> >>  sree_at_chess@yahoo.com
> >>  > > .INVALID>
> >>  > > > > wrote:
> >>  > > > > >
> >>  > > > > > Heron will continue to live long.
> >>  > > > > > It has it's own place in the stream processing world among
> other
> >>  > > > > competing technologies.The ever increasing data has stretched
> >>  > > competitions
> >>  > > > > to the limits of breaking.
> >>  > > > > >
> >>  > > > > > In addition:
> >>  > > > > > In production at the creating company and others around the
> >>  > > world.Best
> >>  > > > > open source alternative to Google Dataflow, from the recent
> talks.
> >>  > > > > > Higher freedom to customizations, makes it attractive for
> >>  > innovation.
> >>  > > > > > 27 continuous monthly meetups.
> >>  > > > > > Slack is active.Mailing lists are active.
> >>  > > > > > 455 meetup members and counting.40 linkedin group members and
> >>  > > counting.
> >>  > > > > >
> >>  > > > > > All of these, just by a few bunch of us.
> >>  > > > > >
> >>  > > > > > It is too early for 'retirement' talk, IMHO.
> >>  > > > > > Let's focus on, making it to TLP.
> >>  > > > > > Taking one task or part of it at a time.
> >>  > > > > >
> >>  > > > > >
> >>  > > > > > Thank you./Sree
> >>  > > > > >
> >>  > > > > > On Thursday, October 15, 2020, 11:00:10 AM PDT, H W <
> >>  > > > > huijun.wu.2010@gmail.com> wrote:
> >>  > > > > >
> >>  > > > > > The community size and activity look steady rather than
> >>  dwindling.
> >>  > > The
> >>  > > > > > heronstreaming slack is still active. The
> >>  > > > > conversations/meetups/discussions
> >>  > > > > > keep going well.
> >>  > > > > > As for 'retirement' I think that would be premature
> >>  > > > > >
> >>  > > > > > On Thu, Oct 15, 2020 at 10:29 AM Ning Wang <
> wangninggm@gmail.com
> >>  >
> >>  > > wrote:
> >>  > > > > >
> >>  > > > > >> Hmm.
> >>  > > > > >>
> >>  > > > > >> Community isn't very active, but there are still works
> going on
> >>  > > (python,
> >>  > > > > >> k8s/helm, etc) and a few users relying on the project. IMO
> it is
> >>  > too
> >>  > > > > early
> >>  > > > > >> to retire.
> >>  > > > > >>
> >>  > > > > >>
> >>  > > > > >>
> >>  > > > > >> On Wed, Oct 14, 2020 at 5:57 PM Josh Fischer <
> >>  josh@joshfischer.io
> >>  > >
> >>  > > > > wrote:
> >>  > > > > >>
> >>  > > > > >>> Hi All,
> >>  > > > > >>>
> >>  > > > > >>> It seems the community is dwindling for Heron. I think it
> is
> >>  time
> >>  > > to
> >>  > > > > >> start
> >>  > > > > >>> a discussion on retiring the podling.
> >>  > > > > >>>
> >>  > > > > >>> Thoughts?
> >>  > > > > >>>
> >>  > > > > >>> - Josh
> >>  > > > > >>>
> >>  > > > > >>
> >>  > > > >
> >>  > > > >
> >>  > > >
> >>  > >
> >>  > >
> >>  >
> >>