You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cheng Zhou <zh...@gmail.com> on 2011/05/21 02:27:11 UTC
How to create document objects in our case
Hi,
I have a large number of XML files to be indexed by Lucene. All the files
share similar structure as below:
<Group id="abc" member="cde" blah blah ....>
<Subgroup id="abc1" member ="fgh" blah blah ...>
<Subgroup id="abc2" member ="fgh" blah blah ...>
<Subgroup id="abc3" member ="fgh" blah blah ...>
......
</Group>
Things to be noted are:
The root element of Group has 30 or so attributes, and it usually has over
2000 Subgroup elements, which in turn also have more than 20 attributes.
I want to create one Document object which holds the contents of the Group
element, and one Document object which holds all the Subgroup elements.
Here are my challenges however:
1. How many fields are advised for a Document to be indexed by Lucene? Will
over 30 fields (for the Group element) be too many?
2. How to create a Document object and fields for holding all the Subgroup
elements? Is this a good way to think of?
3. How can I link the Document object of the Group element to the Document
object of all the Subgroup elements?
Please note that I intend to use such two Document objects to achieve the
group while I don't know whether it is a good solution or not. I am open to
using more than two Documents to do the job, but I don't know how to connect
all the objects in Lucene.
Many thanks!
Re: How to create document objects in our case
Posted by Michael McCandless <lu...@mikemccandless.com>.
You're welcome!
Mike
http://blog.mikemccandless.com
On Sun, May 22, 2011 at 9:20 AM, zhoucheng2008 <zh...@gmail.com> wrote:
> Great, thanks Mike.
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Sunday, May 22, 2011 8:09 PM
> To: java-user@lucene.apache.org
> Subject: Re: How to create document objects in our case
>
> Norms is how Lucene records what the apriori boost is for each
> docXfield. This boost is the product of per-field boost, per-doc
> boost (both of which your app would set when it creates the doc), as
> well as the "length normalization" Lucene's default similarity applies
> (shorter docs have higher boost).
>
> It's a quantized float, encoded as 8 bits.
>
> If your fields tend not to vary in length much, and you don't use
> boosting yourself, and you are worried about RAM, you should omit the
> norms. (Field has a method to omit norms).
>
> Not sure when nested docs will be available but I hope soon... even
> so, once it's committed, it won't be released until the next release
> (though you can use it on the trunk/3.x tip as well, once it's
> committed).
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Sun, May 22, 2011 at 7:49 AM, zhoucheng2008 <zh...@gmail.com>
> wrote:
>> Mike, thanks for reply.
>>
>> Can you please elaborate a little bit more on " If you don't need norms
>> (don't boost, lengths don't vary much or you
>> don't care to have field length impact scoring) you can omit norms"?
>>
>> When do you expect the handling of nested document will be applicable?
>>
>> Cheng
>>
>>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Sunday, May 22, 2011 6:58 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: How to create document objects in our case
>>
>> 30 fields is fine, but if they are all indexed you should watch out
>> for memory usage. Ie, norms require 1 byte per doc per indexed field.
>> If you don't need norms (don't boost, lengths don't vary much or you
>> don't care to have field length impact scoring) you can omit norms.
>>
>> The relationship b/w Group and Subgroup is not something Lucene can do
>> today -- all docs are "independent" in a Lucene index.
>>
>> So.. you can either "denormalize", meaning duplicate all Group fields
>> onto each subgroup, or you can check out
>> https://issues.apache.org/jira/browse/LUCENE-2454 which I think adds
>> exactly what you need. It's not yet committed, but we are finally
>> starting to make progress towards that, eg
>> https://issues.apache.org/jira/browse/LUCENE-3112
>>
>> Mike
>>
>> http://blog.mikemccandless.com
>>
>> On Fri, May 20, 2011 at 8:27 PM, Cheng Zhou <zh...@gmail.com>
> wrote:
>>> Hi,
>>>
>>> I have a large number of XML files to be indexed by Lucene. All the files
>>> share similar structure as below:
>>>
>>> <Group id="abc" member="cde" blah blah ....>
>>> <Subgroup id="abc1" member ="fgh" blah blah ...>
>>> <Subgroup id="abc2" member ="fgh" blah blah ...>
>>> <Subgroup id="abc3" member ="fgh" blah blah ...>
>>> ......
>>> </Group>
>>>
>>> Things to be noted are:
>>>
>>> The root element of Group has 30 or so attributes, and it usually has
> over
>>> 2000 Subgroup elements, which in turn also have more than 20 attributes.
>>>
>>> I want to create one Document object which holds the contents of the
> Group
>>> element, and one Document object which holds all the Subgroup elements.
>>>
>>> Here are my challenges however:
>>>
>>> 1. How many fields are advised for a Document to be indexed by Lucene?
>> Will
>>> over 30 fields (for the Group element) be too many?
>>>
>>> 2. How to create a Document object and fields for holding all the
> Subgroup
>>> elements? Is this a good way to think of?
>>>
>>> 3. How can I link the Document object of the Group element to the
> Document
>>> object of all the Subgroup elements?
>>>
>>> Please note that I intend to use such two Document objects to achieve the
>>> group while I don't know whether it is a good solution or not. I am open
>> to
>>> using more than two Documents to do the job, but I don't know how to
>> connect
>>> all the objects in Lucene.
>>>
>>> Many thanks!
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: How to create document objects in our case
Posted by zhoucheng2008 <zh...@gmail.com>.
Great, thanks Mike.
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Sunday, May 22, 2011 8:09 PM
To: java-user@lucene.apache.org
Subject: Re: How to create document objects in our case
Norms is how Lucene records what the apriori boost is for each
docXfield. This boost is the product of per-field boost, per-doc
boost (both of which your app would set when it creates the doc), as
well as the "length normalization" Lucene's default similarity applies
(shorter docs have higher boost).
It's a quantized float, encoded as 8 bits.
If your fields tend not to vary in length much, and you don't use
boosting yourself, and you are worried about RAM, you should omit the
norms. (Field has a method to omit norms).
Not sure when nested docs will be available but I hope soon... even
so, once it's committed, it won't be released until the next release
(though you can use it on the trunk/3.x tip as well, once it's
committed).
Mike
http://blog.mikemccandless.com
On Sun, May 22, 2011 at 7:49 AM, zhoucheng2008 <zh...@gmail.com>
wrote:
> Mike, thanks for reply.
>
> Can you please elaborate a little bit more on " If you don't need norms
> (don't boost, lengths don't vary much or you
> don't care to have field length impact scoring) you can omit norms"?
>
> When do you expect the handling of nested document will be applicable?
>
> Cheng
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Sunday, May 22, 2011 6:58 PM
> To: java-user@lucene.apache.org
> Subject: Re: How to create document objects in our case
>
> 30 fields is fine, but if they are all indexed you should watch out
> for memory usage. Ie, norms require 1 byte per doc per indexed field.
> If you don't need norms (don't boost, lengths don't vary much or you
> don't care to have field length impact scoring) you can omit norms.
>
> The relationship b/w Group and Subgroup is not something Lucene can do
> today -- all docs are "independent" in a Lucene index.
>
> So.. you can either "denormalize", meaning duplicate all Group fields
> onto each subgroup, or you can check out
> https://issues.apache.org/jira/browse/LUCENE-2454 which I think adds
> exactly what you need. It's not yet committed, but we are finally
> starting to make progress towards that, eg
> https://issues.apache.org/jira/browse/LUCENE-3112
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Fri, May 20, 2011 at 8:27 PM, Cheng Zhou <zh...@gmail.com>
wrote:
>> Hi,
>>
>> I have a large number of XML files to be indexed by Lucene. All the files
>> share similar structure as below:
>>
>> <Group id="abc" member="cde" blah blah ....>
>> <Subgroup id="abc1" member ="fgh" blah blah ...>
>> <Subgroup id="abc2" member ="fgh" blah blah ...>
>> <Subgroup id="abc3" member ="fgh" blah blah ...>
>> ......
>> </Group>
>>
>> Things to be noted are:
>>
>> The root element of Group has 30 or so attributes, and it usually has
over
>> 2000 Subgroup elements, which in turn also have more than 20 attributes.
>>
>> I want to create one Document object which holds the contents of the
Group
>> element, and one Document object which holds all the Subgroup elements.
>>
>> Here are my challenges however:
>>
>> 1. How many fields are advised for a Document to be indexed by Lucene?
> Will
>> over 30 fields (for the Group element) be too many?
>>
>> 2. How to create a Document object and fields for holding all the
Subgroup
>> elements? Is this a good way to think of?
>>
>> 3. How can I link the Document object of the Group element to the
Document
>> object of all the Subgroup elements?
>>
>> Please note that I intend to use such two Document objects to achieve the
>> group while I don't know whether it is a good solution or not. I am open
> to
>> using more than two Documents to do the job, but I don't know how to
> connect
>> all the objects in Lucene.
>>
>> Many thanks!
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to create document objects in our case
Posted by Michael McCandless <lu...@mikemccandless.com>.
Norms is how Lucene records what the apriori boost is for each
docXfield. This boost is the product of per-field boost, per-doc
boost (both of which your app would set when it creates the doc), as
well as the "length normalization" Lucene's default similarity applies
(shorter docs have higher boost).
It's a quantized float, encoded as 8 bits.
If your fields tend not to vary in length much, and you don't use
boosting yourself, and you are worried about RAM, you should omit the
norms. (Field has a method to omit norms).
Not sure when nested docs will be available but I hope soon... even
so, once it's committed, it won't be released until the next release
(though you can use it on the trunk/3.x tip as well, once it's
committed).
Mike
http://blog.mikemccandless.com
On Sun, May 22, 2011 at 7:49 AM, zhoucheng2008 <zh...@gmail.com> wrote:
> Mike, thanks for reply.
>
> Can you please elaborate a little bit more on " If you don't need norms
> (don't boost, lengths don't vary much or you
> don't care to have field length impact scoring) you can omit norms"?
>
> When do you expect the handling of nested document will be applicable?
>
> Cheng
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Sunday, May 22, 2011 6:58 PM
> To: java-user@lucene.apache.org
> Subject: Re: How to create document objects in our case
>
> 30 fields is fine, but if they are all indexed you should watch out
> for memory usage. Ie, norms require 1 byte per doc per indexed field.
> If you don't need norms (don't boost, lengths don't vary much or you
> don't care to have field length impact scoring) you can omit norms.
>
> The relationship b/w Group and Subgroup is not something Lucene can do
> today -- all docs are "independent" in a Lucene index.
>
> So.. you can either "denormalize", meaning duplicate all Group fields
> onto each subgroup, or you can check out
> https://issues.apache.org/jira/browse/LUCENE-2454 which I think adds
> exactly what you need. It's not yet committed, but we are finally
> starting to make progress towards that, eg
> https://issues.apache.org/jira/browse/LUCENE-3112
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Fri, May 20, 2011 at 8:27 PM, Cheng Zhou <zh...@gmail.com> wrote:
>> Hi,
>>
>> I have a large number of XML files to be indexed by Lucene. All the files
>> share similar structure as below:
>>
>> <Group id="abc" member="cde" blah blah ....>
>> <Subgroup id="abc1" member ="fgh" blah blah ...>
>> <Subgroup id="abc2" member ="fgh" blah blah ...>
>> <Subgroup id="abc3" member ="fgh" blah blah ...>
>> ......
>> </Group>
>>
>> Things to be noted are:
>>
>> The root element of Group has 30 or so attributes, and it usually has over
>> 2000 Subgroup elements, which in turn also have more than 20 attributes.
>>
>> I want to create one Document object which holds the contents of the Group
>> element, and one Document object which holds all the Subgroup elements.
>>
>> Here are my challenges however:
>>
>> 1. How many fields are advised for a Document to be indexed by Lucene?
> Will
>> over 30 fields (for the Group element) be too many?
>>
>> 2. How to create a Document object and fields for holding all the Subgroup
>> elements? Is this a good way to think of?
>>
>> 3. How can I link the Document object of the Group element to the Document
>> object of all the Subgroup elements?
>>
>> Please note that I intend to use such two Document objects to achieve the
>> group while I don't know whether it is a good solution or not. I am open
> to
>> using more than two Documents to do the job, but I don't know how to
> connect
>> all the objects in Lucene.
>>
>> Many thanks!
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: How to create document objects in our case
Posted by zhoucheng2008 <zh...@gmail.com>.
Mike, thanks for reply.
Can you please elaborate a little bit more on " If you don't need norms
(don't boost, lengths don't vary much or you
don't care to have field length impact scoring) you can omit norms"?
When do you expect the handling of nested document will be applicable?
Cheng
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Sunday, May 22, 2011 6:58 PM
To: java-user@lucene.apache.org
Subject: Re: How to create document objects in our case
30 fields is fine, but if they are all indexed you should watch out
for memory usage. Ie, norms require 1 byte per doc per indexed field.
If you don't need norms (don't boost, lengths don't vary much or you
don't care to have field length impact scoring) you can omit norms.
The relationship b/w Group and Subgroup is not something Lucene can do
today -- all docs are "independent" in a Lucene index.
So.. you can either "denormalize", meaning duplicate all Group fields
onto each subgroup, or you can check out
https://issues.apache.org/jira/browse/LUCENE-2454 which I think adds
exactly what you need. It's not yet committed, but we are finally
starting to make progress towards that, eg
https://issues.apache.org/jira/browse/LUCENE-3112
Mike
http://blog.mikemccandless.com
On Fri, May 20, 2011 at 8:27 PM, Cheng Zhou <zh...@gmail.com> wrote:
> Hi,
>
> I have a large number of XML files to be indexed by Lucene. All the files
> share similar structure as below:
>
> <Group id="abc" member="cde" blah blah ....>
> <Subgroup id="abc1" member ="fgh" blah blah ...>
> <Subgroup id="abc2" member ="fgh" blah blah ...>
> <Subgroup id="abc3" member ="fgh" blah blah ...>
> ......
> </Group>
>
> Things to be noted are:
>
> The root element of Group has 30 or so attributes, and it usually has over
> 2000 Subgroup elements, which in turn also have more than 20 attributes.
>
> I want to create one Document object which holds the contents of the Group
> element, and one Document object which holds all the Subgroup elements.
>
> Here are my challenges however:
>
> 1. How many fields are advised for a Document to be indexed by Lucene?
Will
> over 30 fields (for the Group element) be too many?
>
> 2. How to create a Document object and fields for holding all the Subgroup
> elements? Is this a good way to think of?
>
> 3. How can I link the Document object of the Group element to the Document
> object of all the Subgroup elements?
>
> Please note that I intend to use such two Document objects to achieve the
> group while I don't know whether it is a good solution or not. I am open
to
> using more than two Documents to do the job, but I don't know how to
connect
> all the objects in Lucene.
>
> Many thanks!
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to create document objects in our case
Posted by Michael McCandless <lu...@mikemccandless.com>.
30 fields is fine, but if they are all indexed you should watch out
for memory usage. Ie, norms require 1 byte per doc per indexed field.
If you don't need norms (don't boost, lengths don't vary much or you
don't care to have field length impact scoring) you can omit norms.
The relationship b/w Group and Subgroup is not something Lucene can do
today -- all docs are "independent" in a Lucene index.
So.. you can either "denormalize", meaning duplicate all Group fields
onto each subgroup, or you can check out
https://issues.apache.org/jira/browse/LUCENE-2454 which I think adds
exactly what you need. It's not yet committed, but we are finally
starting to make progress towards that, eg
https://issues.apache.org/jira/browse/LUCENE-3112
Mike
http://blog.mikemccandless.com
On Fri, May 20, 2011 at 8:27 PM, Cheng Zhou <zh...@gmail.com> wrote:
> Hi,
>
> I have a large number of XML files to be indexed by Lucene. All the files
> share similar structure as below:
>
> <Group id="abc" member="cde" blah blah ....>
> <Subgroup id="abc1" member ="fgh" blah blah ...>
> <Subgroup id="abc2" member ="fgh" blah blah ...>
> <Subgroup id="abc3" member ="fgh" blah blah ...>
> ......
> </Group>
>
> Things to be noted are:
>
> The root element of Group has 30 or so attributes, and it usually has over
> 2000 Subgroup elements, which in turn also have more than 20 attributes.
>
> I want to create one Document object which holds the contents of the Group
> element, and one Document object which holds all the Subgroup elements.
>
> Here are my challenges however:
>
> 1. How many fields are advised for a Document to be indexed by Lucene? Will
> over 30 fields (for the Group element) be too many?
>
> 2. How to create a Document object and fields for holding all the Subgroup
> elements? Is this a good way to think of?
>
> 3. How can I link the Document object of the Group element to the Document
> object of all the Subgroup elements?
>
> Please note that I intend to use such two Document objects to achieve the
> group while I don't know whether it is a good solution or not. I am open to
> using more than two Documents to do the job, but I don't know how to connect
> all the objects in Lucene.
>
> Many thanks!
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org