Indexes without Map/Reduce

[Peter Marron <Pe...@trilliumsoftware.com>]

Hi,

Hope that this is to the correct list. Apologies if not.

I am using Hive 0.11.0 and Hadoop 1.0.4.

My goal is to get my Hive queries running without Map/Reduce
but using my custom indexes. To this end I have been building Hive version 13 from source
and working through the sources to see what I can do.

I can see that the non-M/R path through Hive splits off really early.
I can see that in SemanticAnalyzer.java if it determines that a FetchTask
is sufficient for the query then the genMapRedTasks method returns really
early and it never gets near the code that uses indexes.

I have also followed the code through the index code and I can see that in
IndexWhereProcessor.java an index can insert a "index query" task
to run before the main query. (By also calling the
queryContext  setIndexInputFormat and setIndexIntermediateFile
methods it can redirect the main query to pick up the data generated by the index.)

So I can see two approaches to achieve my goal.


1)      I can modify the FetchTask path to support the use of indexes.


2)      I can allow the query to start down the Map/Reduce path and then

I can arrange for my index code to trash the original query completely and

replace it  with a query that will run as a FetchTask that will do what I want.

Of course there are pros and cons to both of these approaches.


1)      This approach has the advantage that I don't need to change the

current index path at all and so there's much less likely that I will

damage it. However I will probably end up replicating some of the

existing index code, which is not desirable. Also I am not sufficiently

au fait with the Hive code to feel confident that I would make such

a major change in the way that a real Hive developer might.



2)      This approach has the advantage that I am building on top of the

existing index infrastructure and so I probably will end up writing

less code. However it means that my queries will run once as Map

Reduce and again as FetchTasks which will make them slower than

I would like. The approach is also more complicated than I would like.

And I don't really know how cleanly I can "abort" the initial query and

replace it with a FetchTask. (if, indeed, this is possible.)

Obviously at some point I would like for my changes to get submitted
back into the main Hive source and so I want maximize the chances that
they will be viewed positively.

Does anyone have any opinions or advice to offer?

Regards,

Peter Marron
Senior Developer
Trillium Software, A Harte Hanks Company
Theale Court, 1st Floor, 11-13 High Street
Theale
RG7 5AH
+44 (0) 118 940 7609 office
+44 (0) 118 940 7699 fax
[https://4b2685446389bc779b46-5f66fbb59518cc4fcae8900db28267f5.ssl.cf2.rackcdn.com/trillium.png]<http://www.trilliumsoftware.com/>
trilliumsoftware.com<http://www.trilliumsoftware.com/> / linkedin<http://www.linkedin.com/company/17710> / twitter<https://twitter.com/trilliumsw> / facebook<http://www.facebook.com/HarteHanks>