You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Garbage <ga...@gmx.de> on 2012/06/08 13:49:02 UTC
[ApacheDS] how can a partition return more than one result ?
I learned a lot about the implementation of custom partitions and see the changes in the API from search and lookup returning an Entry in 1.5.x and an EntryFilteringCursor later on.
I even was able to change an existing partition (shame on me: based on the 1.5.5 example, but I will switch to 2.0 soon) in a way that lets it return a fake group object that was created in my custom POJO.
But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
Re: [ApacheDS] how can a partition return more than one result ?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 6/11/12 3:21 PM, Garbage a écrit :
>
> Von meinem iPad gesendet
>
>
>>> But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
>> The idea is to use a Cursor that maps around the partition and fetch the entries one by one.
>>
>> The way the server works is that based on your filter, you select the right index to use to fetch the entries. There are may possibilities here :
>> - first, you may have to do a full scan (the filter is not selective enough, for instance). In this case, you don't use any index, you just use the MasterTable to get the entries. Now, for each entry you fetch, you'll have to filter them to see if it's a valid entry - or not.
>> - or you can select an index. You will fetch the index elements, and for each of them, fetch the associated entry. Once done, you can check against the filter if the entry is valid - or not
>>
>> In any case, the cursor is your friend here : it maps the next() operation on top of your index.
>>
>> Now, if your Partition is a Btree, it's easier, as the AbstractBTreePartition class already handles everyting for you. If you don't inherit from this Abstract class, then it's way more complicated. I'll suggest you have a look at the AbstractBTreePartition to get a clue about how we process a search over a BTree based partition.
> If I got you right I am to create my own cursor object which retrieves and stores the array with the items I want to deliver. The cursor then implements the next and get method in order to send the array one by one. But where is the cursor being called ?
It's called at the top level, as we write the entries one by one,
waiting for the socket to be ready for a new write. The cursor does not
have to do anything but to fetch the next entry when requested.
> I tried to base my own partition on the abstract btree partition but what am I supposed to return from getRootId and getDefaultId and even worse from convertAndInit ?
The RootId is pretty simple : it's 0L.
The DefaultId is always 1L too.
The convertAndInit methid is pretty stupid (and I do think we will get
rid of it soon). It basically transform an index from a "generic" form
to a usable form. The rational is that when we initialize the server, we
don't yet know the kind of backend we will use for a partition, so we
created a dumb index containing the configuration needed to initialize
the index in all the case. Then we initialize the backend, and we now
can create the real index (I mean, the JdbmIndex if we have a
JdbmPartition, an AvlIndex for AvlPartition, and an AcmeIndex for your
AcmePartition).
Now, we have pretty much simplified the process, because we always know
what kind of index we will have to create, so we create them
immediately. Sadly, there is still one special index that requires some
extra effort, this is the RdnIndex.
So, for instance, the JdbmPartition uses this kind of method :
protected Index<?> convertAndInit( Index<?> index ) throws Exception
{
JdbmIndex<?> jdbmIndex;
// first, deal with the RdnIndex which is a bit specific. We will create
a new one using the configuration put into the generic JdbmIndex
if ( index.getAttributeId().equals(
ApacheSchemaConstants.APACHE_RDN_AT_OID ) )
{
jdbmIndex = new JdbmRdnIndex();
jdbmIndex.setAttributeId(
ApacheSchemaConstants.APACHE_RDN_AT_OID );
jdbmIndex.setCacheSize( index.getCacheSize() );
jdbmIndex.setNumDupLimit( JdbmIndex.DEFAULT_DUPLICATE_LIMIT );
jdbmIndex.setWkDirPath( index.getWkDirPath() );
}
else if ( index instanceof JdbmIndex<?> )
{
// Then for each other index, as they are alreadt plain JdbmIndex, just
use them and inject the workpath into it
jdbmIndex = ( JdbmIndex<?> ) index;
if ( jdbmIndex.getWkDirPath() == null )
{
jdbmIndex.setWkDirPath( partitionPath );
}
}
else
{
// Should never occur...
LOG.debug( "Supplied index {} is not a JdbmIndex. "
+ "Will create new JdbmIndex using copied configuration
parameters.", index );
jdbmIndex = new JdbmIndex( index.getAttributeId(), true );
jdbmIndex.setCacheSize( index.getCacheSize() );
jdbmIndex.setNumDupLimit( JdbmIndex.DEFAULT_DUPLICATE_LIMIT );
jdbmIndex.setWkDirPath( index.getWkDirPath() );
}
// last thing, we just initialize the index.
jdbmIndex.init( schemaManager,
schemaManager.lookupAttributeTypeRegistry( index.getAttributeId() ) );
return jdbmIndex;
}
You should basically do the exact same thing.
>
> Sorry for the beginner level questions but ApacheDS is quite complex and new to me.
meh... It's also complex to me, except that I'm working on it for now 7
years :/
Btw, we are likely to change many things to the partition in the next
few months, as we have some problem with concurrency, so keep tuned.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: [ApacheDS] how can a partition return more than one result ?
Posted by Garbage <ga...@gmx.de>.
Von meinem iPad gesendet
>> But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
> The idea is to use a Cursor that maps around the partition and fetch the entries one by one.
>
> The way the server works is that based on your filter, you select the right index to use to fetch the entries. There are may possibilities here :
> - first, you may have to do a full scan (the filter is not selective enough, for instance). In this case, you don't use any index, you just use the MasterTable to get the entries. Now, for each entry you fetch, you'll have to filter them to see if it's a valid entry - or not.
> - or you can select an index. You will fetch the index elements, and for each of them, fetch the associated entry. Once done, you can check against the filter if the entry is valid - or not
>
> In any case, the cursor is your friend here : it maps the next() operation on top of your index.
>
> Now, if your Partition is a Btree, it's easier, as the AbstractBTreePartition class already handles everyting for you. If you don't inherit from this Abstract class, then it's way more complicated. I'll suggest you have a look at the AbstractBTreePartition to get a clue about how we process a search over a BTree based partition.
If I got you right I am to create my own cursor object which retrieves and stores the array with the items I want to deliver. The cursor then implements the next and get method in order to send the array one by one. But where is the cursor being called ?
I tried to base my own partition on the abstract btree partition but what am I supposed to return from getRootId and getDefaultId and even worse from convertAndInit ?
Sorry for the beginner level questions but ApacheDS is quite complex and new to me.
Re: [ApacheDS] how can a partition return more than one result ?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 6/8/12 2:05 PM, Garbage a écrit :
>
> Am 08.06.2012 um 13:58 schrieb Emmanuel Lécharny<el...@gmail.com>:
>
>> Le 6/8/12 1:49 PM, Garbage a écrit :
>>> I learned a lot about the implementation of custom partitions and see the changes in the API from search and lookup returning an Entry in 1.5.x and an EntryFilteringCursor later on.
>>> I even was able to change an existing partition (shame on me: based on the 1.5.5 example, but I will switch to 2.0 soon) in a way that lets it return a fake group object that was created in my custom POJO.
>>>
>>> But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
>> The idea is to use a Cursor that maps around the partition and fetch the entries one by one.
>>
>> The way the server works is that based on your filter, you select the right index to use to fetch the entries. There are may possibilities here :
>> - first, you may have to do a full scan (the filter is not selective enough, for instance). In this case, you don't use any index, you just use the MasterTable to get the entries. Now, for each entry you fetch, you'll have to filter them to see if it's a valid entry - or not.
>> - or you can select an index. You will fetch the index elements, and for each of them, fetch the associated entry. Once done, you can check against the filter if the entry is valid - or not
>>
>> In any case, the cursor is your friend here : it maps the next() operation on top of your index.
>>
>> Now, if your Partition is a Btree, it's easier, as the AbstractBTreePartition class already handles everyting for you. If you don't inherit from this Abstract class, then it's way more complicated. I'll suggest you have a look at the AbstractBTreePartition to get a clue about how we process a search over a BTree based partition.
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
> Thanks, I will investigate this. What a pity, I am able to map searches to string arrays containing the name of groups I want to return. But I understand why you don't support arrays directly, I will somehow manage to map to the Bree example. And if not I will show up here again ;-)
If you already have an array, you just have to create your own cursor
wraping it, maintaining the current index, and the next() call will
simply fetch the next entry in the arry, incrementing the pointer. This
is pretty simple to implement, I think.
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: [ApacheDS] how can a partition return more than one result ?
Posted by Garbage <ga...@gmx.de>.
Am 08.06.2012 um 13:58 schrieb Emmanuel Lécharny <el...@gmail.com>:
> Le 6/8/12 1:49 PM, Garbage a écrit :
>> I learned a lot about the implementation of custom partitions and see the changes in the API from search and lookup returning an Entry in 1.5.x and an EntryFilteringCursor later on.
>> I even was able to change an existing partition (shame on me: based on the 1.5.5 example, but I will switch to 2.0 soon) in a way that lets it return a fake group object that was created in my custom POJO.
>>
>> But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
> The idea is to use a Cursor that maps around the partition and fetch the entries one by one.
>
> The way the server works is that based on your filter, you select the right index to use to fetch the entries. There are may possibilities here :
> - first, you may have to do a full scan (the filter is not selective enough, for instance). In this case, you don't use any index, you just use the MasterTable to get the entries. Now, for each entry you fetch, you'll have to filter them to see if it's a valid entry - or not.
> - or you can select an index. You will fetch the index elements, and for each of them, fetch the associated entry. Once done, you can check against the filter if the entry is valid - or not
>
> In any case, the cursor is your friend here : it maps the next() operation on top of your index.
>
> Now, if your Partition is a Btree, it's easier, as the AbstractBTreePartition class already handles everyting for you. If you don't inherit from this Abstract class, then it's way more complicated. I'll suggest you have a look at the AbstractBTreePartition to get a clue about how we process a search over a BTree based partition.
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
Thanks, I will investigate this. What a pity, I am able to map searches to string arrays containing the name of groups I want to return. But I understand why you don't support arrays directly, I will somehow manage to map to the Bree example. And if not I will show up here again ;-)
Re: [ApacheDS] how can a partition return more than one result ?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 6/8/12 1:49 PM, Garbage a écrit :
> I learned a lot about the implementation of custom partitions and see the changes in the API from search and lookup returning an Entry in 1.5.x and an EntryFilteringCursor later on.
> I even was able to change an existing partition (shame on me: based on the 1.5.5 example, but I will switch to 2.0 soon) in a way that lets it return a fake group object that was created in my custom POJO.
>
> But I am only able to return ONE entry, I didn't find or understand the concept how MULTIPLE entries can be returned. Can someone show me the right direction ?
The idea is to use a Cursor that maps around the partition and fetch the
entries one by one.
The way the server works is that based on your filter, you select the
right index to use to fetch the entries. There are may possibilities here :
- first, you may have to do a full scan (the filter is not selective
enough, for instance). In this case, you don't use any index, you just
use the MasterTable to get the entries. Now, for each entry you fetch,
you'll have to filter them to see if it's a valid entry - or not.
- or you can select an index. You will fetch the index elements, and for
each of them, fetch the associated entry. Once done, you can check
against the filter if the entry is valid - or not
In any case, the cursor is your friend here : it maps the next()
operation on top of your index.
Now, if your Partition is a Btree, it's easier, as the
AbstractBTreePartition class already handles everyting for you. If you
don't inherit from this Abstract class, then it's way more complicated.
I'll suggest you have a look at the AbstractBTreePartition to get a clue
about how we process a search over a BTree based partition.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com