Using Lucene Index for Business Intelligence / Analytics

[Saurabh Dani <sa...@danicorp.com>]

				I don't have a lot of experience with reporting tools and how data is stored by high priced tools which use OLAP and other similar storage types but we needed a solution for drill down reports and searching within large application logs / web server logs.

So, we decided to build a reporting / analytics solution using Lucene. The product is almost ready and we are doing closed beta testing for now for reporting & searching in Web Server logs with our small set of beta customers. I am not here to promote the product or drop the URL, but wanted to get an opinion from this group on using Lucene as the data store for Business Intelligence / Analytics.

If implemented correctly, is there any reason why a Lucene based data store cannot compete with the high priced reporting / data analytics products for drill down reports? I know about a lot of Lucene advantages which I can talk about with customers, but what can be the downside of using Lucene / competitive advantages in other products which we should be aware of?

Thanks
Saurabh Dani 

Re: Using Lucene Index for Business Intelligence / Analytics

[Tomi NA <he...@gmail.com>]

On 8/31/06, Saurabh Dani <sa...@danicorp.com> wrote:
>
>
>                                 I don't have a lot of experience with reporting tools and how data is stored by high priced tools which use OLAP and other similar storage types but we needed a solution for drill down reports and searching within large application logs / web server logs.
>
> So, we decided to build a reporting / analytics solution using Lucene. The product is almost ready and we are doing closed beta testing for now for reporting & searching in Web Server logs with our small set of beta customers. I am not here to promote the product or drop the URL, but wanted to get an opinion from this group on using Lucene as the data store for Business Intelligence / Analytics.
>
> If implemented correctly, is there any reason why a Lucene based data store cannot compete with the high priced reporting / data analytics products for drill down reports? I know about a lot of Lucene advantages which I can talk about with customers, but what can be the downside of using Lucene / competitive advantages in other products which we should be aware of?

It doesn't sound promising to me.
>From my own (true, not too rich) experience, search engine has as much
to do with business inteligence as e.g. text processing.
I mean, yes, you can probably draw an analogy of an OLAP drill-down to
searching withing search results, but that's about it...reporting and
analysis tools are fundamentaly different from a search engine in that
they work with highly structured data and leverage that structure to
do various calculations the results of which are then presented to the
user. A search engine, on the other hand, is basically a tool designed
to handle tons of unstructured or badly structured (i.e. structured
for the human reader) data.
In conclusion, you can do great things with a search engine, but BI in
general or reporting in particular is not one of them. Take a look at
bizgres, pentaho (mondrian, jrubik/jpivot, kettle, weka), and
jasperreports to get a better idea of what I'm talking about.

Just another opinion, anyway...

t.n.a.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org