Replication

[Andy Armstrong <an...@tagish.com>]

Hello,

Firstly thanks folks for Xindice; it seems to provide a load of 
functionality that we're looking for.

Secondly: has there been any thought about supporting replication 
between xindice servers? If not then it's something we'd be interested 
in working on.

-- 
Andy Armstrong, Tagish

Re: Full text index, Re: Replication

[Andy Armstrong <an...@tagish.com>]

Vadim Gritsenko wrote:

[snip]
> May be some tweaks would be required to index manager to make it 
> properly work with this type of index, and that should be it.

Wow. That's a lot more of a start than I'd hoped for. Excellent, thanks. 
I'm hoping we'll be able to start work on it next week.

-- 
Andy Armstrong, Tagish

Full text index, Re: Replication

[Vadim Gritsenko <va...@reverycodes.com>]

Andy Armstrong wrote:

> The full text indexing thing actually takes precedence at the moment. 
> I'll be having a look at that next week if anyone has any bright ideas :)


Actually, I was thinking about adding it too.

Full text index can be built on top of Lucene, by creating new type of 
index. Currently, Xindice has NameIndexer and ValueIndexer, 
implementations of the org.apache.xindice.core.indexer.Indexer 
interface. New implementation should be added, can be called 
TextIndexer, I guess (or, FullTextIndexer). Indexer interface already 
defines what index patter should look like:

     * getPattern returns the pattern recognized by this Indexer.  Patterns
     * must be in the form of (elem|*)[@(attr|*)] to tell the IndexManager
     * which element types to send to it, so for example:
     * <pre>
     *    contact@name  Indexes all contacts by name attribute
     *    memo          Indexes the text of all memo elements
     *    contact@*     Indexes all contact attributes
     *    *@name        Indexes the name attribute for all elements
     *    *             Indexes the text of all elements
     *    *@*           Indexes all attributes of all elements
     * </pre>

And index style is already defined:

    String STYLE_NODENAME = "Node:Name";
    String STYLE_NODEVALUE = "Node:Value";
    String STYLE_FULLTEXT = "Node:FullText";

May be some tweaks would be required to index manager to make it 
properly work with this type of index, and that should be it.

Vadim

Re: Replication

[Upayavira <uv...@upaya.co.uk>]

Andy Armstrong wrote:

> Vadim Gritsenko wrote:
>
>>> Firstly thanks folks for Xindice; it seems to provide a load of 
>>> functionality that we're looking for.
>>>
>>> Secondly: has there been any thought about supporting replication 
>>> between xindice servers? If not then it's something we'd be 
>>> interested in working on.
>>
>>
>> No, there is no replication now. How do you want to approach it?
>
>
> Slowly :)
>
> Before doing anything else we need to alter our product to use Xindice 
> - at the moment it's just storing documents on the filesystem. Once 
> that's done I'll be thinking about the best ways to:
>
>  * build full text indexes of the documents in a collection
>  * replicate between Xindice servers
>
> We currently solve the full text problem by loading our whole document 
> tree into memory at startup and building an index at that time. We 
> then periodically scan the filesystem for updated documents and 
> reload/reindex them accordingly so we need to be able to replace that 
> mechanism without merely loading all the documents from Xindice at 
> startup.
>
> The replication mechanism I'm most familiar with is the one Lotus 
> Domino uses. The main problem if you're doing peer to peer replication 
> is how to resolve update conflicts and I suspect that the 'right' 
> solution depends to a significant extent on the application so my 
> feeling is that that's the area that needs the most consideration.
>
> The full text indexing thing actually takes precedence at the moment. 
> I'll be having a look at that next week if anyone has any bright ideas :)
>
The Apache Lucene project is a full text search engine. It is included 
within Cocoon for doing full text searches of Cocoon based web sites 
(see the Cocoon Lucene block in Cocoon under src/blocks/lucene for the 
code). With the right interface, allowing Lucene to crawl documents in 
your Xindice database, it could then build for you a full text index 
that you can query at your leasure. Probably something that could be 
quite useful as an option for all XIndice users.

Regards, Upayavira

Re: Replication

[Andy Armstrong <an...@tagish.com>]

Vadim Gritsenko wrote:

>> Firstly thanks folks for Xindice; it seems to provide a load of 
>> functionality that we're looking for.
>>
>> Secondly: has there been any thought about supporting replication 
>> between xindice servers? If not then it's something we'd be interested 
>> in working on.
> 
> No, there is no replication now. How do you want to approach it?

Slowly :)

Before doing anything else we need to alter our product to use Xindice - 
at the moment it's just storing documents on the filesystem. Once that's 
done I'll be thinking about the best ways to:

  * build full text indexes of the documents in a collection
  * replicate between Xindice servers

We currently solve the full text problem by loading our whole document 
tree into memory at startup and building an index at that time. We then 
periodically scan the filesystem for updated documents and 
reload/reindex them accordingly so we need to be able to replace that 
mechanism without merely loading all the documents from Xindice at startup.

The replication mechanism I'm most familiar with is the one Lotus Domino 
uses. The main problem if you're doing peer to peer replication is how 
to resolve update conflicts and I suspect that the 'right' solution 
depends to a significant extent on the application so my feeling is that 
that's the area that needs the most consideration.

The full text indexing thing actually takes precedence at the moment. 
I'll be having a look at that next week if anyone has any bright ideas :)

-- 
Andy Armstrong, Tagish

Re: Replication

[Vadim Gritsenko <va...@reverycodes.com>]

Andy Armstrong wrote:

> Hello,
>
> Firstly thanks folks for Xindice; it seems to provide a load of 
> functionality that we're looking for.
>
> Secondly: has there been any thought about supporting replication 
> between xindice servers? If not then it's something we'd be interested 
> in working on.


No, there is no replication now. How do you want to approach it?

Vadim