Document path representation as multivalued fields: sensible?

[Marc <se...@busy-byte.org>]

Hi there,

I am running an application comprising approx. 20 million documents. To 
make these documents searchable, I decided to give SOLR a try and feed 
meta-information about my documents into SOLR using a Python script. 
This all works fine. My question is there less a technical one, but 
rather a structural/strategic one.

In my document collection, documents can have parent-documents, and in 
particular not only one parent document, but potentially several parent 
documents. Each document is identified uniquely by an ID value. Child 
documents refer to their parents using a multi-value field 'parent' 
which holds the parent's ID values.

I am interested in the paths that lead from leaf-documents (documents 
that only have parents, but no further children) back to the root 
document. My idea was to add any parent document of a child document 
(also those further away than the immediate parents, so grandparent and 
grand-grand-... parent documents) into this multi-value parent property.

Later, I want to be able to pick any document X from my 20 million 
documents and efficiently determine the set of documents of which X is a 
parent. I.e., all documents that have an entry of X's ID somewhere in 
their parent field.

* Is my strategy of structuring my documents by means of the multi-value 
property sensible?
* Does SOLR provide better methods (that I'm not aware of) to achieve 
the same?
* Will this perform properly? Or is my structuring method likely to keel 
over at some stage if the number of documents keeps growing?

Best,
Marc

Re: Document path representation as multivalued fields: sensible?

[Marc <se...@busy-byte.org>]

Hi there,

I saw that SOLR's got a special graph query parser. If I understand 
things correctly, that parser could comput the documents paths at query 
time. Instead of storing all parent node references in each child 
element, storing only the immediate parents should be sufficient with 
this parser.

Can someone enlighten me about the time such a path computation would 
take? Considering up to 20 levels of hierarchy (wild guess), would it be 
beneficial to use the graph query parser over some application layer 
logic as outlined in my initial posting below? Can anyone share 
experiences with the graph query parser?

Best,
Marc

Am 22.11.2024 10:54, schrieb ufuk yılmaz:
> I don’t like multivalued fields much because they don’t play nice with
> docValues which enable many cool features I like about Solr. They also
> don’t match indexes (find docs that have this value on the 3rd
> position). But don’t take this as a suggestion not to use them, they
> have their use.
> 
> Check
> https://solr.apache.org/guide/solr/latest/indexing-guide/tokenizers.html#path-hierarchy-tokenizer
> as an alternative to see if it can help with your case
> 
> -ufuk yilmaz
> 
> 
> 
> —
> 
>> On Nov 21, 2024, at 22:13, Marc <se...@busy-byte.org> wrote:
>> 
>> Hi there,
>> 
>> I am running an application comprising approx. 20 million documents. 
>> To make these documents searchable, I decided to give SOLR a try and 
>> feed meta-information about my documents into SOLR using a Python 
>> script. This all works fine. My question is there less a technical 
>> one, but rather a structural/strategic one.
>> 
>> In my document collection, documents can have parent-documents, and in 
>> particular not only one parent document, but potentially several 
>> parent documents. Each document is identified uniquely by an ID value. 
>> Child documents refer to their parents using a multi-value field 
>> 'parent' which holds the parent's ID values.
>> 
>> I am interested in the paths that lead from leaf-documents (documents 
>> that only have parents, but no further children) back to the root 
>> document. My idea was to add any parent document of a child document 
>> (also those further away than the immediate parents, so grandparent 
>> and grand-grand-... parent documents) into this multi-value parent 
>> property.
>> 
>> Later, I want to be able to pick any document X from my 20 million 
>> documents and efficiently determine the set of documents of which X is 
>> a parent. I.e., all documents that have an entry of X's ID somewhere 
>> in their parent field.
>> 
>> * Is my strategy of structuring my documents by means of the 
>> multi-value property sensible?
>> * Does SOLR provide better methods (that I'm not aware of) to achieve 
>> the same?
>> * Will this perform properly? Or is my structuring method likely to 
>> keel over at some stage if the number of documents keeps growing?
>> 
>> Best,
>> Marc

Re: Document path representation as multivalued fields: sensible?

[ufuk yılmaz <uy...@vivaldi.net.INVALID>]

I don’t like multivalued fields much because they don’t play nice with docValues which enable many cool features I like about Solr. They also don’t match indexes (find docs that have this value on the 3rd position). But don’t take this as a suggestion not to use them, they have their use. 

Check https://solr.apache.org/guide/solr/latest/indexing-guide/tokenizers.html#path-hierarchy-tokenizer as an alternative to see if it can help with your case

-ufuk yilmaz



—

> On Nov 21, 2024, at 22:13, Marc <se...@busy-byte.org> wrote:
> 
> Hi there,
> 
> I am running an application comprising approx. 20 million documents. To make these documents searchable, I decided to give SOLR a try and feed meta-information about my documents into SOLR using a Python script. This all works fine. My question is there less a technical one, but rather a structural/strategic one.
> 
> In my document collection, documents can have parent-documents, and in particular not only one parent document, but potentially several parent documents. Each document is identified uniquely by an ID value. Child documents refer to their parents using a multi-value field 'parent' which holds the parent's ID values.
> 
> I am interested in the paths that lead from leaf-documents (documents that only have parents, but no further children) back to the root document. My idea was to add any parent document of a child document (also those further away than the immediate parents, so grandparent and grand-grand-... parent documents) into this multi-value parent property.
> 
> Later, I want to be able to pick any document X from my 20 million documents and efficiently determine the set of documents of which X is a parent. I.e., all documents that have an entry of X's ID somewhere in their parent field.
> 
> * Is my strategy of structuring my documents by means of the multi-value property sensible?
> * Does SOLR provide better methods (that I'm not aware of) to achieve the same?
> * Will this perform properly? Or is my structuring method likely to keel over at some stage if the number of documents keeps growing?
> 
> Best,
> Marc