You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Li Bing <lb...@gmail.com> on 2009/08/10 18:40:28 UTC
Lucene.NET without DB
Hi, all,
Is it possible to use Lucene.NET only without DB? What about the performance?
I appreciate so much for your help!
LBLabs
Re: Lucene.NET without DB
Posted by Michael Barbarelli <mb...@gmail.com>.
Yes, of course. Unless by "DB" you mean Lucene index. Then, no.
On Mon, Aug 10, 2009 at 5:40 PM, Li Bing <lb...@gmail.com> wrote:
> Hi, all,
>
> Is it possible to use Lucene.NET only without DB? What about the
> performance?
>
> I appreciate so much for your help!
>
> LBLabs
>
Re: Lucene.NET without DB
Posted by Li Bing <lb...@gmail.com>.
Wagner,
Thanks so much for your suggestions! I think your explanation is great.
Best,
LB
On Wed, Aug 12, 2009 at 1:24 AM, Wagner Ignacio Pinto
Junior<wa...@hotmail.com> wrote:
>
>
> Hi Li,
> Wayne,
>
> I think the decision about to use or not to use an RDBMS should be based on features you need.
>
> Wayne is right to warn you that Lucene is not meant to replace RDBMS, but if you do not need RDBMS features I think it is ok to only use Lucene to store data. (Someone correct me if I'm mistaken) :)
>
> I do use Lucene and SQL Server, and both have their own role, Lucene for full text search and SQL Server for relational data, and I keep both in sync dealing with CRUD.
>
> If you go for Lucene alone, be careful with updates to your data, because you will need to copy all fields manually, something you would not need to worry with a RDBMS update.
>
> Regards,
> Wagner Junior
>
> > Date: Tue, 11 Aug 2009 13:27:07 +0800
>> Subject: Re: Lucene.NET without DB
>> From: lblabs@gmail.com
>> To: lucene-net-user@incubator.apache.org
>>
>> Dear Wayne,
>>
>> I might not clarify what I am doing.
>>
>> I didn't establish a traditional Web site. What I am doing is a
>> searching engine with some new techniques. I need to crawl pages from
>> some Web sites and manage them on my site. I think when users search
>> the pages, Lucene can support for sure. However, when I manage the
>> crawled links locally myself, a database, such as MySQL, must be good
>> enough. This is my initial design.
>>
>> But if so, my site must have two modules to manage data. One is Lucene
>> and the other is MySQL. Is it necessary? I even found that MySQL had a
>> bug, such as "Invalid attempt to access a field before calling
>> Read()". So if Lucene can do both the above work, I might not use
>> MySQL any more. How do you think about that?
>>
>> Thanks so much!
>> LB
>>
>> On Tue, Aug 11, 2009 at 1:14 PM, Wayne
>> Douglas<co...@googlemail.com> wrote:
>> > Li, U can store HTML on disk and lucene can index that. But to think that a
>> > db makes a website cumbersome, for all but the most basic sites, unless
>> > you're doing some radical thinking, you're digging yourself into a hole.
>> >
>> > Do you have some more detail about the site you're trying to build and the
>> > implementation you're thinking about using?
>> >
>> > -----------------------------
>> > e-wayne <at> isit.gd
>> > t-07525 424 882
>> >
>> > On 11 Aug 2009, at 06:03 AM, Li Bing <lb...@gmail.com> wrote:
>> >
>> >> Thanks so much for your responses!
>> >>
>> >> Originally my system use MySQL to save some non-full-text searching
>> >> data and use Lucene to manage my crawled Web pages. However, I think
>> >> MySQL makes my system cumbersome. So I decide to manage all of my data
>> >> by Lucene no matter if the data is full-text searching based or not.
>> >>
>> >> According to your emails, I think the above approach is feasible,
>> >> right? Thanks again!
>> >>
>> >> Best,
>> >> LB
>> >>
>> >> On Tue, Aug 11, 2009 at 3:27 AM, Michael
>> >> Barbarelli<mb...@gmail.com> wrote:
>> >>>
>> >>> Sorry if my last message was terse , Li. The beauty of Lucene is that
>> >>> you can basically index from almost any data source imaginable. You
>> >>> will just have to do the work of parsing through that data yourself.
>> >>>
>> >>> On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
>> >>>>
>> >>>> Hi, all,
>> >>>>
>> >>>> Is it possible to use Lucene.NET only without DB? What about the
>> >>>> performance?
>> >>>>
>> >>>> I appreciate so much for your help!
>> >>>>
>> >>>> LBLabs
>> >>>>
>> >>>
>> >
>
RE: Lucene.NET without DB
Posted by Wagner Ignacio Pinto Junior <wa...@hotmail.com>.
Hi Li,
Wayne,
I think the decision about to use or not to use an RDBMS should be based on features you need.
Wayne is right to warn you that Lucene is not meant to replace RDBMS, but if you do not need RDBMS features I think it is ok to only use Lucene to store data. (Someone correct me if I'm mistaken) :)
I do use Lucene and SQL Server, and both have their own role, Lucene for full text search and SQL Server for relational data, and I keep both in sync dealing with CRUD.
If you go for Lucene alone, be careful with updates to your data, because you will need to copy all fields manually, something you would not need to worry with a RDBMS update.
Regards,
Wagner Junior
> Date: Tue, 11 Aug 2009 13:27:07 +0800
> Subject: Re: Lucene.NET without DB
> From: lblabs@gmail.com
> To: lucene-net-user@incubator.apache.org
>
> Dear Wayne,
>
> I might not clarify what I am doing.
>
> I didn't establish a traditional Web site. What I am doing is a
> searching engine with some new techniques. I need to crawl pages from
> some Web sites and manage them on my site. I think when users search
> the pages, Lucene can support for sure. However, when I manage the
> crawled links locally myself, a database, such as MySQL, must be good
> enough. This is my initial design.
>
> But if so, my site must have two modules to manage data. One is Lucene
> and the other is MySQL. Is it necessary? I even found that MySQL had a
> bug, such as "Invalid attempt to access a field before calling
> Read()". So if Lucene can do both the above work, I might not use
> MySQL any more. How do you think about that?
>
> Thanks so much!
> LB
>
> On Tue, Aug 11, 2009 at 1:14 PM, Wayne
> Douglas<co...@googlemail.com> wrote:
> > Li, U can store HTML on disk and lucene can index that. But to think that a
> > db makes a website cumbersome, for all but the most basic sites, unless
> > you're doing some radical thinking, you're digging yourself into a hole.
> >
> > Do you have some more detail about the site you're trying to build and the
> > implementation you're thinking about using?
> >
> > -----------------------------
> > e-wayne <at> isit.gd
> > t-07525 424 882
> >
> > On 11 Aug 2009, at 06:03 AM, Li Bing <lb...@gmail.com> wrote:
> >
> >> Thanks so much for your responses!
> >>
> >> Originally my system use MySQL to save some non-full-text searching
> >> data and use Lucene to manage my crawled Web pages. However, I think
> >> MySQL makes my system cumbersome. So I decide to manage all of my data
> >> by Lucene no matter if the data is full-text searching based or not.
> >>
> >> According to your emails, I think the above approach is feasible,
> >> right? Thanks again!
> >>
> >> Best,
> >> LB
> >>
> >> On Tue, Aug 11, 2009 at 3:27 AM, Michael
> >> Barbarelli<mb...@gmail.com> wrote:
> >>>
> >>> Sorry if my last message was terse , Li. The beauty of Lucene is that
> >>> you can basically index from almost any data source imaginable. You
> >>> will just have to do the work of parsing through that data yourself.
> >>>
> >>> On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
> >>>>
> >>>> Hi, all,
> >>>>
> >>>> Is it possible to use Lucene.NET only without DB? What about the
> >>>> performance?
> >>>>
> >>>> I appreciate so much for your help!
> >>>>
> >>>> LBLabs
> >>>>
> >>>
> >
Re: Lucene.NET without DB
Posted by Li Bing <lb...@gmail.com>.
Dear Wayne,
I might not clarify what I am doing.
I didn't establish a traditional Web site. What I am doing is a
searching engine with some new techniques. I need to crawl pages from
some Web sites and manage them on my site. I think when users search
the pages, Lucene can support for sure. However, when I manage the
crawled links locally myself, a database, such as MySQL, must be good
enough. This is my initial design.
But if so, my site must have two modules to manage data. One is Lucene
and the other is MySQL. Is it necessary? I even found that MySQL had a
bug, such as "Invalid attempt to access a field before calling
Read()". So if Lucene can do both the above work, I might not use
MySQL any more. How do you think about that?
Thanks so much!
LB
On Tue, Aug 11, 2009 at 1:14 PM, Wayne
Douglas<co...@googlemail.com> wrote:
> Li, U can store HTML on disk and lucene can index that. But to think that a
> db makes a website cumbersome, for all but the most basic sites, unless
> you're doing some radical thinking, you're digging yourself into a hole.
>
> Do you have some more detail about the site you're trying to build and the
> implementation you're thinking about using?
>
> -----------------------------
> e-wayne <at> isit.gd
> t-07525 424 882
>
> On 11 Aug 2009, at 06:03 AM, Li Bing <lb...@gmail.com> wrote:
>
>> Thanks so much for your responses!
>>
>> Originally my system use MySQL to save some non-full-text searching
>> data and use Lucene to manage my crawled Web pages. However, I think
>> MySQL makes my system cumbersome. So I decide to manage all of my data
>> by Lucene no matter if the data is full-text searching based or not.
>>
>> According to your emails, I think the above approach is feasible,
>> right? Thanks again!
>>
>> Best,
>> LB
>>
>> On Tue, Aug 11, 2009 at 3:27 AM, Michael
>> Barbarelli<mb...@gmail.com> wrote:
>>>
>>> Sorry if my last message was terse , Li. The beauty of Lucene is that
>>> you can basically index from almost any data source imaginable. You
>>> will just have to do the work of parsing through that data yourself.
>>>
>>> On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
>>>>
>>>> Hi, all,
>>>>
>>>> Is it possible to use Lucene.NET only without DB? What about the
>>>> performance?
>>>>
>>>> I appreciate so much for your help!
>>>>
>>>> LBLabs
>>>>
>>>
>
Re: Lucene.NET without DB
Posted by Wayne Douglas <co...@googlemail.com>.
Li, U can store HTML on disk and lucene can index that. But to think
that a db makes a website cumbersome, for all but the most basic
sites, unless you're doing some radical thinking, you're digging
yourself into a hole.
Do you have some more detail about the site you're trying to build and
the implementation you're thinking about using?
-----------------------------
e-wayne <at> isit.gd
t-07525 424 882
On 11 Aug 2009, at 06:03 AM, Li Bing <lb...@gmail.com> wrote:
> Thanks so much for your responses!
>
> Originally my system use MySQL to save some non-full-text searching
> data and use Lucene to manage my crawled Web pages. However, I think
> MySQL makes my system cumbersome. So I decide to manage all of my data
> by Lucene no matter if the data is full-text searching based or not.
>
> According to your emails, I think the above approach is feasible,
> right? Thanks again!
>
> Best,
> LB
>
> On Tue, Aug 11, 2009 at 3:27 AM, Michael
> Barbarelli<mb...@gmail.com> wrote:
>> Sorry if my last message was terse , Li. The beauty of Lucene is
>> that
>> you can basically index from almost any data source imaginable. You
>> will just have to do the work of parsing through that data yourself.
>>
>> On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
>>> Hi, all,
>>>
>>> Is it possible to use Lucene.NET only without DB? What about the
>>> performance?
>>>
>>> I appreciate so much for your help!
>>>
>>> LBLabs
>>>
>>
Re: Lucene.NET without DB
Posted by Li Bing <lb...@gmail.com>.
Thanks so much for your responses!
Originally my system use MySQL to save some non-full-text searching
data and use Lucene to manage my crawled Web pages. However, I think
MySQL makes my system cumbersome. So I decide to manage all of my data
by Lucene no matter if the data is full-text searching based or not.
According to your emails, I think the above approach is feasible,
right? Thanks again!
Best,
LB
On Tue, Aug 11, 2009 at 3:27 AM, Michael
Barbarelli<mb...@gmail.com> wrote:
> Sorry if my last message was terse , Li. The beauty of Lucene is that
> you can basically index from almost any data source imaginable. You
> will just have to do the work of parsing through that data yourself.
>
> On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
>> Hi, all,
>>
>> Is it possible to use Lucene.NET only without DB? What about the performance?
>>
>> I appreciate so much for your help!
>>
>> LBLabs
>>
>
Re: Lucene.NET without DB
Posted by Michael Barbarelli <mb...@gmail.com>.
Sorry if my last message was terse , Li. The beauty of Lucene is that
you can basically index from almost any data source imaginable. You
will just have to do the work of parsing through that data yourself.
On Monday, August 10, 2009, Li Bing <lb...@gmail.com> wrote:
> Hi, all,
>
> Is it possible to use Lucene.NET only without DB? What about the performance?
>
> I appreciate so much for your help!
>
> LBLabs
>
RE: Lucene.NET without DB
Posted by Franklin Simmons <fs...@sccmediaserver.com>.
Absolutely.
For example if you were indexing web pages, you could index the text of the web page in a field named 'content' with Field.Store.NO and Field.Index.TOKENIZED, and the URL of the web page in a field named 'resource' with Field.Store.YES and Field.Index.NO. When processing search results, you can load the 'resource' fields to fetch the URLs.
-----Original Message-----
From: Li Bing [mailto:lblabs@gmail.com]
Sent: Monday, August 10, 2009 12:40 PM
To: lucene-net-user@incubator.apache.org
Subject: Lucene.NET without DB
Hi, all,
Is it possible to use Lucene.NET only without DB? What about the performance?
I appreciate so much for your help!
LBLabs