[jr3] Search index in content

[Jukka Zitting <ju...@gmail.com>]

Hi,

Having the search index stored on the normal file system gives us
great performance and avoids having to tweak Lucene too much to suit
our needs. However, the extra storage mechanism adds all sorts of
consistency, backup and transaction complexities.

Especially with things like the data store it should be feasible to
achieve good enough search performance even if the index is stored
inside the repository either as normal (hidden/protected) JCR nodes or
as special lower level constructs that still leverage our standard
persistence mechanism.

What would we need to do to achieve this, and would the result be
worth the effort?

BR,

Jukka Zitting

Re: [jr3] Search index in content

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Feb 17, 2010 at 4:51 PM, Ard Schrijvers
<a....@onehippo.com> wrote:
> Also still, I really doubt about whether it is feasible to implement
> in a performant way

The key performance problem we faced earlier with non-local search
indexes was the lack of efficient random access read to the index
segments. I believe we should be able to solve that issue with
something like a data store implementation that keeps copies of
frequently used records on local disk and offers direct random access
needed to efficiently implement the Lucene IndexInput API.

BR,

Jukka Zitting

Re: [jr3] Search index in content

[Thomas Müller <th...@day.com>]

> I'd use Lucene to manage it.

There are several problems. One is transactions, another is updating
the index synchronously. Another problem is dependence on Lucene which
is a problem for persistence and clustering.

> I would very much like to avoid inventing our own search index.

I would definitely not use a completely new mechanism. I would re-use
the repository to store the index data.

Regards,
Thomas

Re: [jr3] Search index in content

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Feb 18, 2010 at 8:39 AM, Thomas Müller <th...@day.com> wrote:
> The property/value indexes [...] Currently, we use Apache Lucene for this,
> but I wouldn't. I would keep those indexes within the repository.

I would very much like to avoid inventing our own search index. Even
after moving the index into the repository, I'd use Lucene to manage
it.

BR,

Jukka Zitting

Re: [jr3] Search index in content

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Thu, Feb 18, 2010 at 8:39 AM, Thomas Müller <th...@day.com> wrote:
> The property/value indexes are for property values, node names, paths,
> node references, and so on. ....
> ...Currently, we use Apache Lucene for this, but I wouldn't. I
> would keep those indexes within the repository.
>
> ...The fulltext index is (potentially) slow, specially fulltext
> extraction. Therefore, fulltext index should be done asynchronously if
> it takes too long....

I love this idea of separating the two kinds of indexes, having the
fulltext "eventually indexed" might be good enough, with interfaces to
find out about the status of the indexing queue.

I am involved in the IKS project (http://iks-project.eu/) where we
envision new types of indexing/search for content-based applications,
and in this perspective it might make sense to be able to add more
indexing methods.

So as we're dreaming aloud, my ideal view of Jackrabbit
indexing/search would be:

1. The "structural index" (your first type) is managed by Jackrabbit,
doesn't require configuration, behaves like a database index
(synchronous, transactional, stored in repository, etc.)

2. The "standard fulltext index" uses Lucene, large items are queued
for eventual indexing, can be delegated to a separate cluster,
configurable as to what to index and what not, can be disabled, etc.
Ideally stored in repository.

3. Additional "custom external indexes" can be configured, work like
the Lucene index but using external components (a la Solr for example,
RESTful indexing engines). Not sure how the JCR query syntax can
address those, that's a different problem.

-Bertrand

Re: [jr3] Search index in content

[Alexander Klimetschek <ak...@day.com>]

On Thu, Feb 18, 2010 at 10:23, Ard Schrijvers <a....@onehippo.com> wrote:
>> The fulltext index is (potentially) slow, specially fulltext
>> extraction. Therefore, fulltext index should be done asynchronously if
>
> would this be in line with the spec?

AFAIU the spec allows this and we already do this: long-running full
text extractors are queued and their result is added asynchronously,
after the session has been committed.

Regards,
Alex

-- 
Alexander Klimetschek
alexander.klimetschek@day.com

Re: [jr3] Search index in content

[Ard Schrijvers <a....@onehippo.com>]

On Thu, Feb 18, 2010 at 8:39 AM, Thomas Müller <th...@day.com> wrote:

>
> The fulltext index is (potentially) slow, specially fulltext
> extraction. Therefore, fulltext index should be done asynchronously if

would this be in line with the spec?

> it takes too long. Also, in a clustered environment, at least text
> extraction should only be done in one cluster node. I would still use
> Apache Tika and Apache Lucene for this.

Especially pdf extraction can kill the performance of an entire
cluster. As pdfs can be part of a document at our structure, where it
needs to be nodescope indexed every time the document is saved again,
we use an approach to store as binary (to use the DataStore) version
an extracted version of the pdf and index this extracted version: Only
one node in the cluster will now do the extraction, only one user is
blocked. The other nodes just index the extracted text version, which
is quite fast. Not sure if we should have this kind of option part of
JR

regards Ard

>
> Regards,
> Thomas
>

Re: [jr3] Search index in content

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Feb 18, 2010 at 2:11 PM, Thomas Müller <th...@day.com> wrote:
> In any case, keeping the index and the persistence in the
> same storage simplifies transactional persistence a lot.

Agreed. I believe we can do this equally well (if not better) with
Lucene as with any homegrown indexing mechanism.

> A microkernel that relies on Apache Lucene even for simple
> property/value indexes is not an option in my view.

Agreed. I wouldn't put search features in the microkernel.

BR,

Jukka Zitting

Re: [jr3] Search index in content

[Thomas Müller <th...@day.com>]

Hi,

Property/value indexes: We anyway will have to implement some kind of
database persistence. Databases support transactional indexes. We
could use those instead of using Lucene. Or we could store the index
in JCR nodes (which is part of the large repository b-tree). Indexes
in databases are stored in exactly the same way.

In any case, keeping the index and the persistence in the same storage
simplifies transactional persistence a lot.

A microkernel that relies on Apache Lucene even for simple
property/value indexes is not an option in my view.

Regards,
Thomas

Re: [jr3] Search index in content

[Marcel Reutegger <ma...@gmx.net>]

On Thu, Feb 18, 2010 at 13:22, Alexander Klimetschek <ak...@day.com> wrote:
> On Thu, Feb 18, 2010 at 08:39, Thomas Müller <th...@day.com> wrote:
>> For me, there are two kinds of indexes: the property/value indexes,
>> and the fulltext index.
>
> The difficulty is that JCR queries can easily mix the use of both
> indexes, ie. search for a certain node type (property/value index) and
> a fulltext search of its contents.

this would potentially become a major problem because it is nearly
impossible to efficiently evaluate expressions with distinct index
sources. e.g. consider the following simple query:

//element(*, nt:resource)[jcr:contains(., 'frequent-word')

this statement can be evaluated quite efficiently in lucene, but would
probably become an issue when the type constraint and the fulltext
expression come from distinct index structures. the only common
identifier is the node id, which is not necessarily the primary id in
both indices.

regards
 marcel

> I am sure we can implement this
> mixed lookup ourselves, but with Lucene we already get it for free. I
> would stick with Lucene, also because it is proven, active and faster
> improving than we could do on our own.
>
> Regards,
> Alex
>
> --
> Alexander Klimetschek
> alexander.klimetschek@day.com
>

Re: [jr3] Search index in content

[Alexander Klimetschek <ak...@day.com>]

On Thu, Feb 18, 2010 at 08:39, Thomas Müller <th...@day.com> wrote:
> For me, there are two kinds of indexes: the property/value indexes,
> and the fulltext index.

The difficulty is that JCR queries can easily mix the use of both
indexes, ie. search for a certain node type (property/value index) and
a fulltext search of its contents. I am sure we can implement this
mixed lookup ourselves, but with Lucene we already get it for free. I
would stick with Lucene, also because it is proven, active and faster
improving than we could do on our own.

Regards,
Alex

-- 
Alexander Klimetschek
alexander.klimetschek@day.com

Re: [jr3] Search index in content

[Thomas Müller <th...@day.com>]

Hi,

For me, there are two kinds of indexes: the property/value indexes,
and the fulltext index.

The property/value indexes are for property values, node names, paths,
node references, and so on. Such indexes (or "indices") are relatively
small and fast. In relational databases, those are the secondary
indexes (non-primary-key indexes). Those index updates should be done
synchronously as part of the transaction (maybe even in the transient
space). Currently, we use Apache Lucene for this, but I wouldn't. I
would keep those indexes within the repository.

The fulltext index is (potentially) slow, specially fulltext
extraction. Therefore, fulltext index should be done asynchronously if
it takes too long. Also, in a clustered environment, at least text
extraction should only be done in one cluster node. I would still use
Apache Tika and Apache Lucene for this.

Regards,
Thomas

Re: [jr3] Search index in content

[Adam Foltzer <ac...@gmail.com>]

My project has run into problems along these lines as a result of
clustering. Since our application servers are managed very strictly by a
centralized infrastructure group, we have no guarantees of having the same
filesystem available if we have to bring up another server to meet load or
balance a failure. In that case, the entire repository gets reindexed over
JDBC, which can take quite a while with 15,000 documents.

Of course, I appreciate the performance benefits of having the index
locally, but I wonder if there might be a middle ground that could
substantially help clustered deployments. Perhaps each server could maintain
its own index locally, syncing at intervals with a central datasource. If
stored in a protected JCR node, the existing journaling might work to keep
things current? Then, if a new repository joins the cluster, it could pull
down a ready-to-go index.

No idea if this is feasible, of course, just dreaming out loud :)

Cheers,
Adam Foltzer

Re: [jr3] Search index in content

[Ard Schrijvers <a....@onehippo.com>]

On Wed, Feb 17, 2010 at 4:47 PM, Alexander Klimetschek <ak...@day.com> wrote:
> On Wed, Feb 17, 2010 at 16:22, Jukka Zitting <ju...@gmail.com> wrote:
>> What would we need to do to achieve this, and would the result be
>> worth the effort?

>From the top of my head we should also closely think about the changed
way in a clustered environment, as now, all nodes in a cluster have
their own cluster.

Also still, I really doubt about whether it is feasible to implement
in a performant way

Regards Ard

>
> Some ideas from the top of my head:
>
> A basic problem is that the search index would include itself, so
> maybe one has to have a separate node tree under
> /jcr:system/searchIndex that is not covered by the search, observation
> and/or other things that could lead to "circular" problems. Or
> performance issues, if the involved binaries are touched very often
> and also changed outside of sessions, eg. when a text extractor ran
> later.
>
> One "useful" feature of the search index on the file system is that if
> it gets corrupted somehow, you can delete it and it will be re-built
> upon startup. This should also work if it lies inside the content.
> Thus a broken search index must not break repository startup or it
> must be possible to delete it with a tool w/o requiring a full
> repository start.
>
> Regards,
> Alex
>
> --
> Alexander Klimetschek
> alexander.klimetschek@day.com
>

Re: [jr3] Search index in content

[Ian Boston <ie...@tfd.co.uk>]

On 17 Feb 2010, at 15:47, Alexander Klimetschek wrote:

> Thus a broken search index must not break repository startup or it
> must be possible to delete it with a tool w/o requiring a full
> repository start.

Over in an earlier version of Sakai we have a distributed Lucene Index, where the index is produced by all nodes in the cluster indexing  items and distributing the updates to segments. On one hand this distributes the load well giving a single index over the whole cluster, however, there is increased latency compared to the JR implementation, and, critically, when the index gets corrupted, its often hard to recover. Snapshots are taken real time, but you have to know how far back to go and then reindex. In 24 months of running JR 1.4 and this search index we have had to recover the distributed search index several times, but not had to recover the JR index once, although its become corrupted almost as many times (once or twice max).

The distributed index, uses local file storage and segment update shipping. We did try various types of off machine storage but found them all to be way to slow. IMVHO seek performance is critical to Lucene performance. 

Ian

Re: [jr3] Search index in content

[Alexander Klimetschek <ak...@day.com>]

On Wed, Feb 17, 2010 at 16:22, Jukka Zitting <ju...@gmail.com> wrote:
> What would we need to do to achieve this, and would the result be
> worth the effort?

Some ideas from the top of my head:

A basic problem is that the search index would include itself, so
maybe one has to have a separate node tree under
/jcr:system/searchIndex that is not covered by the search, observation
and/or other things that could lead to "circular" problems. Or
performance issues, if the involved binaries are touched very often
and also changed outside of sessions, eg. when a text extractor ran
later.

One "useful" feature of the search index on the file system is that if
it gets corrupted somehow, you can delete it and it will be re-built
upon startup. This should also work if it lies inside the content.
Thus a broken search index must not break repository startup or it
must be possible to delete it with a tool w/o requiring a full
repository start.

Regards,
Alex

-- 
Alexander Klimetschek
alexander.klimetschek@day.com

Re: [jr3] Search index in content

[Thomas Müller <th...@day.com>]

+1

For simple search a built-in index would help a lot, for example node
names and (some) property values. Each property name could have its
own index. Advantages:

- transactional index updates
- reduced complexity
- reduced number of open files
- allows to implement Jackrabbit in C

I would not try to use that to index binaries (fulltext index), or
re-implement advanced features (ranking, phrase queries,
stemming,...).

Regards,
Thomas