Exciting Use Case for Groovy - Log Analytics

[Gerald Wiltse <je...@gmail.com>]

Summary:
We are going to use Groovy for more-than-trivial log-parsing and
analytics.  The groovy language native functionality seems fairly-well
suited for this, but probably far from purpose-built Query Languages.

Questions:
Is it well-suited enough to simply teach a dev-ops team to write the rules
in native groovy with closures (none of them know groovy)? Or, should I try
to create a SQL-like Query Language within groovy for this (which will be
more familiar to the average sysadmin/dba)?  Is that the idea behind
creating DSL's?  But is that just re-inventing the wheel and overkill for a
small IT company? Should I instead look to integrate one of the countless
new and exciting query/analytics languages into Groovy?  If so, do any make
particular sense?

Requirements:
I know... every answer depends on requirements/context. The JVMs that runs
the scripts each monitor about 15-25 servers, run one copy of the script
for each server every 5 minutes, and return 5-500 log lines each time. We
currently support about 20 of these JVM's, scaling to 50 or 100 in the next
year.  If we are successful, we might make a specialty practice out of this
form of specialized log-parsing as a service, so efficiency, scalability,
modularity, flexibility... all very real factors.


Background:
I'm not sure if much of the Groovy community is aware, but Logicmonitor is
an amazing SAAS based infrastructure monitoring tool that enables users to
write custom Groovy scripts to be embedded and executed for monitoring
purposes. They just added powershell, but it was groovy-only for a few
years now.

I've been using these to do custom monitoring of Oracle-based applications
for a year (mostly JMX and JDBC queries with lots of math and logic). It's
really starting to come together as a contender in the dev-ops monitoring
space.

They've now added more robust Log-File support, enabling digestion of log
files in groovy scripts for the stated purposes of alerting based on what
is returned from the script, this is why I am looking for advise on the
best approach.


Gerald R. Wiltse
jerrywiltse@gmail.com
248-893-9110 (c)
888-248-7095 (p)
888-272-6046 (f)

Re: Exciting Use Case for Groovy - Log Analytics

[Gerald Wiltse <je...@gmail.com>]

Thanks for your response, it's great to have interest!

Indeed the goal is not to create a Groovy-based log analytic tool/platform
and release it as a contender to anything.  I do not see the future there,
as the market is saturated with really good solutions (as you point out).

The question we're really touching on here is:  "How many Groovy developers
would actually do more with logs within their existing Groovy projects if a
new log/data stream library for Groovy was released (besides me). I guess
that's a key question, and one I would love to hear feedback on.  Does it
rate high enough on anybody's chart for attention? I don't know.

Obviously we would, but that's because of a very niche combination of
factors for us. We were 90% there with Logicmonitor + Groovy for
monitoring.  With the need to just add good (but not amazing) log analysis
and alerting, it makes perfect sense.  Still, even in our best case
scenario, we would have to turn to other data analytics platforms for
anything beyond the "IT Operations" type log work we're intending to do.

Having said that (and after reading a lot about building DSL's with
groovy), I believe there's a good argument for the existence of a Groovy
library that provides a bunch of new power and sugar for working with log
files, and querying streams of data.  Although it's pure speculation, I
would even dare to say that a lot of people would make heavy use of such a
library.  What I'm not sure about, is whether there's a good enough
argument for anyone to spend the time to build such a thing. That seems
like a hard argument to make for anything these days with everyone buried
under backlogs.

Regards,
Jerry


Gerald R. Wiltse
jerrywiltse@gmail.com
248-893-9110 (c)
888-248-7095 (p)
888-272-6046 (f)

On Wed, Dec 30, 2015 at 8:56 PM, Aristedes Maniatis <ar...@ish.com.au> wrote:

> On 31/12/2015 8:46am, Gerald Wiltse wrote:
> > We are going to use Groovy for more-than-trivial log-parsing and
> analytics.  The groovy language native functionality seems fairly-well
> suited for this, but probably far from purpose-built Query Languages.
>
> I'm interested in the work you are doing here, but how will your solution
> differ from the elasticsearch + logstash + kibana system which already
> might do a lot of what you are wanting? Don't get me wrong, I'd love to see
> a groovy log parser over the jruby implementation of logstash, but what is
> the value-add in the project you are contemplating compared to what is
> already available?
>
> Happy new year!
> Ari
>
>
> --
> -------------------------->
> Aristedes Maniatis
> ish
> http://www.ish.com.au
> Level 1, 30 Wilson Street Newtown 2042 Australia
> phone +61 2 9550 5001   fax +61 2 9550 4001
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>
>

Re: Exciting Use Case for Groovy - Log Analytics

[Aristedes Maniatis <ar...@ish.com.au>]

On 31/12/2015 8:46am, Gerald Wiltse wrote:
> We are going to use Groovy for more-than-trivial log-parsing and analytics.  The groovy language native functionality seems fairly-well suited for this, but probably far from purpose-built Query Languages.  

I'm interested in the work you are doing here, but how will your solution differ from the elasticsearch + logstash + kibana system which already might do a lot of what you are wanting? Don't get me wrong, I'd love to see a groovy log parser over the jruby implementation of logstash, but what is the value-add in the project you are contemplating compared to what is already available?

Happy new year!
Ari


-- 
-------------------------->
Aristedes Maniatis
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A