You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Rajesh parab <ra...@yahoo.com> on 2006/11/03 04:03:34 UTC
Modelling relational data in Lucene Index?
Hi,
As I understand, Lucene has a flat structure where you can define multiple fields inside the document. There is no relationship between any field.
I would like to enable index based search for some of the components inside relational database. For exmaple, let say "Folder" Object. The Folder object can have relationship with File object. The File object, in turn, can have attributes like is image, is text file, etc. So, the stricture is
Folder -- > File
|
------- > is image, is text file, ......
I would like to enable a search to find a Folder with File of type image. How can we model such relational data inside Lucene index?
Regards,
Rajesh
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modelling relational data in Lucene Index?
Posted by KEGan <kh...@gmail.com>.
Hi,
I am actually doing something what the original poster mentioned.
Previously, I have use Hibernate and Lucene. But I found that for my
particular project my data is quite flat, so in the next version I totally
take out Hibernate (and the complexity with it :)) and use Lucene as the
"main storage".
In this new version, my data is persisted into filesystem simply using
XMLEncoder. Lucene is both used as a text search index, and also provides
reference to the encoded data (.xml) in the filesystem. Everytime data is
added, an entry into Lucene will be made. And I am using RAMDirectory (super
fast), hence if the server ever shutdown/restart, the Lucene in built again
on startup.
This works for my case because my data set is small enough (hey I read
1.1million documents only average about 300MB of lucene index, and I
have
plenty of RAM), my data probably wont reach anywhere near 0.5 million. The
cons are startup will be slow when data increase, but server shouldnt be
down that often.
Is anyone using similar model ? Any pitfall that I should be aware about ?
Thanks.
~KEGan
On 11/6/06, Emmanuel Bernard <em...@hibernate.org> wrote:
>
> I had a quick look at SOLR and DBSight. They seem to achieve a different
> goal than Hibernate Lucene.
> The formers belong to the project box category: you set up a server that
> will handle the search for you. The application will then delegate the
> work to those servers.
> The latter belongs to the framework category: you use it inside your
> Hibernate/EJB 3.0 application to enable an index based search feature.
> To a certain extend, it is the same difference between a Google box and
> Lucene.
>
> You can write some code based on the latter to covers the formers
> features esp the platform abstraction (PHP, .net), but it is probably a
> lot of work and that is not really the point.
> You can write some code based on the formers to enable indexing and
> search of your persistent domain model (persisted through Hibernate),
> but that is probably more work.
>
> Really it is a matter of easing the pain from one side of the problem or
> the other side. I don't see much competition between the 2 approaches,
> they cover different goals.
>
> To specifically answer some of your remarks:
> - yes, you need to write some code to recreate an index. Literally, 6
> lines of code.
> - no, I do not currently cache the searcher because, Hibernate is
> transactional by nature and protect yourself as much as possible from
> read uncommited and other data inconsistencies. I guess I could
> implement some caching capabilities using reader.isCurrent() or
> something equivalent.
> - the ability to split searchers servers from indexers servers is on my
> todo list.
>
> Cheers
>
> Emmanuel
>
>
> Chris Lu wrote:
> > I personally like your effort, but technically I would disagree.
> >
> > The SOLR project, and the project I am working on, DBSight, have an
> > detached approach which is implementation agnostic, no matter if it's
> > java, ruby, php, .net. The return results can be a rendered HTML,
> > JSON, XML. I don't think you can be more flexible than that. If
> > creating an new index takes 5 minutes without any coding, you can
> > create something more creative.
> >
> >> From business side, you don't need to worry about indexing when
> > designing a system. New requirement may come. It's very hard trying to
> > anticipate all the needs.
> >
> > Technically, detached approach gives more flexible on resources like
> > CPU, memory, hard drive. For example, if your index grows large, say
> > 1G, indexing can take hours with merging, I am not sure how compass or
> > hibernate/lucene handles it. Need to re-write code at that time? I
> > actually feel it's a dangerous trap.
> >
> >> I've introduced a session.index() which forces the (re)indexing of the
> >> document
> > So does it mean you need to write some code to fix the index if it's
> > crashed?
> >
> >> What do you mean by multithread safe? The indexing?
> >> the indexing is multithread safe in the Hibernate Lucene integration
> > The indexing can be threadsafe. But will it affect the searching? With
> > many files changing and merging, if you cache the searcher. the
> > searching will have "read passed EOF" exceptions. If you don't cache
> > the searcher, you will loose the built-in caching, FieldCacheImpl, in
> > Lucene.
> >
> >>
> >> The query process?
> >> the query doesn't have to since you query on a give session (aka user
> >> conversation), so no multithread threat here.
> > So you are not caching searcher.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Modelling relational data in Lucene Index?
Posted by Emmanuel Bernard <em...@hibernate.org>.
I had a quick look at SOLR and DBSight. They seem to achieve a different
goal than Hibernate Lucene.
The formers belong to the project box category: you set up a server that
will handle the search for you. The application will then delegate the
work to those servers.
The latter belongs to the framework category: you use it inside your
Hibernate/EJB 3.0 application to enable an index based search feature.
To a certain extend, it is the same difference between a Google box and
Lucene.
You can write some code based on the latter to covers the formers
features esp the platform abstraction (PHP, .net), but it is probably a
lot of work and that is not really the point.
You can write some code based on the formers to enable indexing and
search of your persistent domain model (persisted through Hibernate),
but that is probably more work.
Really it is a matter of easing the pain from one side of the problem or
the other side. I don't see much competition between the 2 approaches,
they cover different goals.
To specifically answer some of your remarks:
- yes, you need to write some code to recreate an index. Literally, 6
lines of code.
- no, I do not currently cache the searcher because, Hibernate is
transactional by nature and protect yourself as much as possible from
read uncommited and other data inconsistencies. I guess I could
implement some caching capabilities using reader.isCurrent() or
something equivalent.
- the ability to split searchers servers from indexers servers is on my
todo list.
Cheers
Emmanuel
Chris Lu wrote:
> I personally like your effort, but technically I would disagree.
>
> The SOLR project, and the project I am working on, DBSight, have an
> detached approach which is implementation agnostic, no matter if it's
> java, ruby, php, .net. The return results can be a rendered HTML,
> JSON, XML. I don't think you can be more flexible than that. If
> creating an new index takes 5 minutes without any coding, you can
> create something more creative.
>
>> From business side, you don't need to worry about indexing when
> designing a system. New requirement may come. It's very hard trying to
> anticipate all the needs.
>
> Technically, detached approach gives more flexible on resources like
> CPU, memory, hard drive. For example, if your index grows large, say
> 1G, indexing can take hours with merging, I am not sure how compass or
> hibernate/lucene handles it. Need to re-write code at that time? I
> actually feel it's a dangerous trap.
>
>> I've introduced a session.index() which forces the (re)indexing of the
>> document
> So does it mean you need to write some code to fix the index if it's
> crashed?
>
>> What do you mean by multithread safe? The indexing?
>> the indexing is multithread safe in the Hibernate Lucene integration
> The indexing can be threadsafe. But will it affect the searching? With
> many files changing and merging, if you cache the searcher. the
> searching will have "read passed EOF" exceptions. If you don't cache
> the searcher, you will loose the built-in caching, FieldCacheImpl, in
> Lucene.
>
>>
>> The query process?
>> the query doesn't have to since you query on a give session (aka user
>> conversation), so no multithread threat here.
> So you are not caching searcher.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modelling relational data in Lucene Index?
Posted by Chris Lu <ch...@gmail.com>.
I personally like your effort, but technically I would disagree.
The SOLR project, and the project I am working on, DBSight, have an
detached approach which is implementation agnostic, no matter if it's
java, ruby, php, .net. The return results can be a rendered HTML,
JSON, XML. I don't think you can be more flexible than that. If
creating an new index takes 5 minutes without any coding, you can
create something more creative.
>From business side, you don't need to worry about indexing when
designing a system. New requirement may come. It's very hard trying to
anticipate all the needs.
Technically, detached approach gives more flexible on resources like
CPU, memory, hard drive. For example, if your index grows large, say
1G, indexing can take hours with merging, I am not sure how compass or
hibernate/lucene handles it. Need to re-write code at that time? I
actually feel it's a dangerous trap.
> I've introduced a session.index() which forces the (re)indexing of the
> document
So does it mean you need to write some code to fix the index if it's crashed?
> What do you mean by multithread safe? The indexing?
> the indexing is multithread safe in the Hibernate Lucene integration
The indexing can be threadsafe. But will it affect the searching? With
many files changing and merging, if you cache the searcher. the
searching will have "read passed EOF" exceptions. If you don't cache
the searcher, you will loose the built-in caching, FieldCacheImpl, in
Lucene.
>
> The query process?
> the query doesn't have to since you query on a give session (aka user
> conversation), so no multithread threat here.
So you are not caching searcher.
--
Chris Lu
-------------------------
Instant Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
On 11/3/06, Emmanuel Bernard <em...@hibernate.org> wrote:
> Hi,
> What exactly are your concerned about the "non-detached" approach (see
> below)?
>
> Chris Lu wrote:
> >
> > I would prefer a detached approach instead of Hibernate or EJB's
> > approach, which is kind of too tightly coupled with any system. How to
> it is probably going to be couple with yours ;-)
> > rebuild if the index is corrupted, or you have a new Analyzer, or
> I've introduced a session.index() which forces the (re)indexing of the
> document
> > schema evolves? How to make it multi-thread safe?
> What do you mean by multithread safe? The indexing?
> the indexing is multithread safe in the Hibernate Lucene integration
>
> The query process?
> the query doesn't have to since you query on a give session (aka user
> conversation), so no multithread threat here.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modelling relational data in Lucene Index?
Posted by Emmanuel Bernard <em...@hibernate.org>.
Hi,
What exactly are your concerned about the "non-detached" approach (see
below)?
Chris Lu wrote:
>
> I would prefer a detached approach instead of Hibernate or EJB's
> approach, which is kind of too tightly coupled with any system. How to
it is probably going to be couple with yours ;-)
> rebuild if the index is corrupted, or you have a new Analyzer, or
I've introduced a session.index() which forces the (re)indexing of the
document
> schema evolves? How to make it multi-thread safe?
What do you mean by multithread safe? The indexing?
the indexing is multithread safe in the Hibernate Lucene integration
The query process?
the query doesn't have to since you query on a give session (aka user
conversation), so no multithread threat here.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modelling relational data in Lucene Index?
Posted by Chris Lu <ch...@gmail.com>.
For this specific question, you can create index on files, search
files that of type image, and from matched files, find the unique
directories(can be done in lucene or you can do it via java).
Of course this does not scale to deeper relationships. Usually you do
need to flattern the database objects in order to use lucene. It's
just trading space for speed.
I would prefer a detached approach instead of Hibernate or EJB's
approach, which is kind of too tightly coupled with any system. How to
rebuild if the index is corrupted, or you have a new Analyzer, or
schema evolves? How to make it multi-thread safe?
--
Chris Lu
-------------------------
Instant Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
On 11/2/06, Mark Miller <ma...@gmail.com> wrote:
> Lucene is probably not the solution if you are looking for a relational
> model. You should be using a database for that. If you want to combine
> Lucene with a relational model, check out Hibernate and the new EJB
> annotations that it supports...there is a cool little Lucene add-on that
> lets you declare fields to be indexed (and how) with annotations.
>
> - Mark
>
> Rajesh parab wrote:
> > Hi,
> >
> > As I understand, Lucene has a flat structure where you can define multiple fields inside the document. There is no relationship between any field.
> >
> > I would like to enable index based search for some of the components inside relational database. For exmaple, let say "Folder" Object. The Folder object can have relationship with File object. The File object, in turn, can have attributes like is image, is text file, etc. So, the stricture is
> >
> > Folder -- > File
> > |
> > ------- > is image, is text file, ......
> >
> >
> > I would like to enable a search to find a Folder with File of type image. How can we model such relational data inside Lucene index?
> >
> > Regards,
> > Rajesh
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modelling relational data in Lucene Index?
Posted by Mark Miller <ma...@gmail.com>.
Lucene is probably not the solution if you are looking for a relational
model. You should be using a database for that. If you want to combine
Lucene with a relational model, check out Hibernate and the new EJB
annotations that it supports...there is a cool little Lucene add-on that
lets you declare fields to be indexed (and how) with annotations.
- Mark
Rajesh parab wrote:
> Hi,
>
> As I understand, Lucene has a flat structure where you can define multiple fields inside the document. There is no relationship between any field.
>
> I would like to enable index based search for some of the components inside relational database. For exmaple, let say "Folder" Object. The Folder object can have relationship with File object. The File object, in turn, can have attributes like is image, is text file, etc. So, the stricture is
>
> Folder -- > File
> |
> ------- > is image, is text file, ......
>
>
> I would like to enable a search to find a Folder with File of type image. How can we model such relational data inside Lucene index?
>
> Regards,
> Rajesh
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org