Searching repeating fields

[Subodh Damle <id...@damle.name>]

Hi all.

We've been using Lucene to index our dynamic data structure and so far 
Lucene has been flexible enough to accommodate our requirements.

Now we have this requirement about searching repeating fields, whose 
implementation is not clear.

Our data records have a dynamic tree-like structure :
e.g. a portion of record will look like:
-company-data
--- financial-data
------revenue-info
--------year
--------amount

For above record portion we create Lucene fields 
"/company-data/financial-data/revenue-info/year" and 
"/company-data/financial-data/revenue-info/amount"

Here, the 'revenue-info' is a repeating node, so we can have records like :
Record 1
---financial-data
------revenue-info
--------year = 2000
--------amount = 1000000
------revenue-info
--------year = 2001
--------amount = 2000000

Record 2
---financial-data
------revenue-info
--------year = 2000
--------amount = 2000000


Now, we need to find records where 'year=2000' and 'amount=2000000' -- 
only those **belonging to same revenue-info node**.
So the search should match Record2 above , but NOT Record1.
Here, simple BooleanQuery using two Terms 
(/company-data/financial-data/revenue-info/year:2000 and  
/company-data/financial-data/revenue-info/amount:2000000)  cannot do 
this - since it will also match Record 1.

A couple of clarifications : each of the nodes (company-data, 
financial-data, revenue-info ... etc ) have physical records in DB and 
are individually accessible.
Also, we generate Lucene queries programmatically , we don't use the 
query parser at all.

We thought of a couple of approaches to implement this but both have 
some limitations :

Approach 1. Generate a separate dummy Document for each 'revenue-info' 
node That way, we can ensure that both fields matched belong to same 
revenue-info node. The 'revenue-info' Document can be then linked to 
actual record Document using some other 'id' field.
This approach would however increase index size (one additional Doc per 
repeating node ). [ We would have a few dozen repeating nodes in each 
record. ]

Approach 2 : Merge values of 'year' and 'amount' into a single field. 
e.g. /company-data/financial-data/revenue-info-dummy-field: = 2000#2000000
However, the problem is , we may need to do range queries for some 
fields -- so for queries like year =2000 AND amount>1000000, we can't 
search this composite field value. So this approach may be unusable for us.

Has someone encountered something similar ? Can it be implemented any 
other way ?

Any suggestion would be greatly appreciated.

-Subodh

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Searching repeating fields

[Paul Elschot <pa...@xs4all.nl>]

On Thursday 22 June 2006 21:18, Subodh Damle wrote:
> Hi all.
> 
> We've been using Lucene to index our dynamic data structure and so far 
> Lucene has been flexible enough to accommodate our requirements.
> 
> Now we have this requirement about searching repeating fields, whose 
> implementation is not clear.
> 
> Our data records have a dynamic tree-like structure :
> e.g. a portion of record will look like:
> -company-data
> --- financial-data
> ------revenue-info
> --------year
> --------amount
> 
> For above record portion we create Lucene fields 
> "/company-data/financial-data/revenue-info/year" and 
> "/company-data/financial-data/revenue-info/amount"
> 
> Here, the 'revenue-info' is a repeating node, so we can have records like :
> Record 1
> ---financial-data
> ------revenue-info
> --------year = 2000
> --------amount = 1000000
> ------revenue-info
> --------year = 2001
> --------amount = 2000000
> 
> Record 2
> ---financial-data
> ------revenue-info
> --------year = 2000
> --------amount = 2000000
> 
> 
> Now, we need to find records where 'year=2000' and 'amount=2000000' -- 
> only those **belonging to same revenue-info node**.

Other than enumerating field names one can also add a seperator
term between the repeating "fields" and use a search a single "field"
by requiring that the separator does not occur in the match,
roughly like:

SpanNotQuery(SpanNearQuery(term1, term2), separatorterm)

where each term is a SpanTermQuery in the Lucene field,
not the same as the repeating "field" above.

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Searching repeating fields

[Chris Hostetter <ho...@fucit.org>]

: Here, the 'revenue-info' is a repeating node, so we can have records like :
: Record 1
: ---financial-data
: ------revenue-info
: --------year = 2000
: --------amount = 1000000
: ------revenue-info
: --------year = 2001
: --------amount = 2000000
:
: Record 2
: ---financial-data
: ------revenue-info
: --------year = 2000
: --------amount = 2000000

: Has someone encountered something similar ? Can it be implemented any
: other way ?

depending on how many financial data blocks you'll have in a given
"record" you can enumerate them in field names, and then search on all of
the possible enumerarations...

Doc#1...
field_revenue-info_1_year = 2000
field_revenue-info_1_amount = 1000000
field_revenue-info_2_year = 2001
field_revenue-info_2_amount = 2000000

Doc#2...
field_revenue-info_1_year = 2000
field_revenue-info_1_amount = 2000000

Query...
  ((+field_revenue-info_1_year:2000 +field_revenue-info_1_amount:[2000000 TO *])
   (+field_revenue-info_2_year:2000 +field_revenue-info_2_amount:[2000000 TO *]))


...but this approach is really only feasible when there is a limited
number of times these "sets' of fields will be repeated, and you know what
the upper bound is at query time.


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org