[DISCUSSION] Roadmap for Chukwa 0.9 or retire Chukwa

[Eric Yang <ey...@apache.org>]

Hi all,

Chukwa has been a fun project to work.  However, Apache is about building a
community where code can thrive to have long lasting Software that everyone
would use.  This brings us to a cross road of contribute and thrive or
retire dead project.  We are at the cross road again, that we can either
united and build a monitoring and log collection system that people would
want to use, or we can retire Chukwa.

The recent rise of ELK stack, or collectd/Grafana stack are providing log
collection and metrics in their own limited approach.  There is so much
more potential for Chukwa because Chukwa uses HDFS to keep data more
resilient than the traditional implementations.

Spark on HBase provides a great window for Chukwa to transport log and
metrics to HBase, and Chukwa can provide the insights of the metrics and
logs by cross referencing data from different data sources.

For Chukwa to provide value to our users, it needs the following features:

1.  Ease of Deployment.  (Integrate with Ambari for deployment)
2.  Better integration with metrics coming from Ambari Metrics.  It might
be possible to replace Ambari Metrics collector with a more robust and
distributed version as Chukwa Agent adaptors.
3.  Spark scripts on top of Spark on HBase to compute aggregate and analyze
data.

YARN provides a really difficult user model to find user application logs.
Chukwa provided search capability give user the power to search for
application logs, and drill down to the actual container log.  However, not
many people know about these features.  It will require Chukwa developers
to show other the way.  This can not be done by one person.  It will take a
community to implement missing features from above points to bring Chukwa
to be mainstream. I am not sure if anyone is still interested in building
out those features, or we should retire Chukwa for good.

I like to offer people to committer access if they can implement one of the
features above.  I like to hear from the community to understand where our
trajectory is going to land.  This will help Apache to decide the direction
of the project.  Thank you all for your input.

regards,
Eric