Index cleanup and improvements

[Emmanuel Lécharny <el...@gmail.com>]

Hi guys,

today, I removed the GenericIndex class, which was used only to create 
the index from the configuration. I just delegated the creation to the 
XXXPartition classes, using an abstract method (createSystemIndex) in 
the AbstractBtreePartition class. it works well. Now, i'm trying to get 
rid of all the normalization done in the index implementation : we 
should *never* normalize there, all the values should already have been 
normalized before (this will save some extra time).

The next step would probably be to get rid of the reverse index from 
almost all the index but the Rdn index (where it's mandatory). That 
means we will have to reorganize the index content, moving the reverse 
table to the RdnIndex (for both the Avl and Jdbm indexes). I'm not sure 
I'll work on that right now.

Two more things :
- it could be a good idea to add a system index for AttributeType, it 
could help us to know which entries will be impacted if we remove an AT
- we should get rid of the oneLevel and subLevel index, and using the 
Rdn index instead. Stefan has already successfully removed them for the 
Hbase partition, there is no reason we should not get rid of them for 
the Jdbm and Avl partitions too.

All in all, removal of those 2 indexes, plus removal of the reverse 
index, plus removal of the spurious normalization could give us better 
performances for the modifications operations.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index cleanup and improvements

[Emmanuel Lécharny <el...@gmail.com>]

Le 3/21/12 2:04 AM, Alex Karasulu a écrit :
> Hi Emm,
>
> Man I really think you're playing with fire here. At least please work in a
> branch in case something goes wrong. These are not simple changes. I don't
> think you can presently remove the reverse index but who knows I might be
> wrong. I remember you tried this before and decided it was not a good idea
> or something.
I have not yet removed anything. And I'm not sure I will do it any time 
soon. I just exposed some of the ideas around the index.
>
> Plus this is going to big time effect Selcuk on a merge. Can we at least
> hold off on this stuff or work a little more incrementally?
Don't worry. I periodically merge trunks into selcuk's branch, to be 
sure the branch is not lagging too much. I merged it just before my 
vacations, two weeks ago, and since then, there was not too many changes.



-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index cleanup and improvements

[Alex Karasulu <ak...@apache.org>]

Hi Emm,

Man I really think you're playing with fire here. At least please work in a
branch in case something goes wrong. These are not simple changes. I don't
think you can presently remove the reverse index but who knows I might be
wrong. I remember you tried this before and decided it was not a good idea
or something.

Plus this is going to big time effect Selcuk on a merge. Can we at least
hold off on this stuff or work a little more incrementally?

Thanks,
Alex

On Tue, Mar 20, 2012 at 7:37 PM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi guys,
>
> today, I removed the GenericIndex class, which was used only to create the
> index from the configuration. I just delegated the creation to the
> XXXPartition classes, using an abstract method (createSystemIndex) in the
> AbstractBtreePartition class. it works well. Now, i'm trying to get rid of
> all the normalization done in the index implementation : we should *never*
> normalize there, all the values should already have been normalized before
> (this will save some extra time).
>
> The next step would probably be to get rid of the reverse index from
> almost all the index but the Rdn index (where it's mandatory). That means
> we will have to reorganize the index content, moving the reverse table to
> the RdnIndex (for both the Avl and Jdbm indexes). I'm not sure I'll work on
> that right now.
>
> Two more things :
> - it could be a good idea to add a system index for AttributeType, it
> could help us to know which entries will be impacted if we remove an AT
> - we should get rid of the oneLevel and subLevel index, and using the Rdn
> index instead. Stefan has already successfully removed them for the Hbase
> partition, there is no reason we should not get rid of them for the Jdbm
> and Avl partitions too.
>
> All in all, removal of those 2 indexes, plus removal of the reverse index,
> plus removal of the spurious normalization could give us better
> performances for the modifications operations.
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Best Regards,
-- Alex