How to provide full text search over a union of one read-only and one mutable graph

[Brandon Sara <br...@collectivemedicaltech.com.INVALID>]

Here is my scenario:
- I have a large read-only set of data and a very small mutable set of data.
- The read-only set of data can be pre-populated into a TDB2 while the mutable set of data will start out empty but will, obviously have data injected overtime during runtime.
- I want one Fuseki service which combines both of these sets of data into a single dataset so that inference can be performed for the mutable data dependent upon the read-only data. (I.E. union them as named graphs into a single dataset).
- I know that I can accomplish the union of the two sets of data as separate named graphs in the final dataset, but I also need full text search on both the read-only and the mutable portions of the final dataset.

How do I accomplish the need for indexing both read-only and mutable data that get combined into a single dataset? Do I create two indexes? Am I able to share indexes? I know that I could create an initial index from the read-only data which could then be updated when changes are made to the mutable data. However, once the read-only data needs updates (which it will from time to time), I need to regenerate the entire index in order to get any changes that weren’t added via an update through Fuseki (which is how the read-only data would be updated, likely a new tdb2 would be generated).

Thanks for the help!
No PHI in Email: PointClickCare and Collective Medical, A PointClickCare Company, policies prohibit sending protected health information (PHI) by email, which may violate regulatory requirements. If sending PHI is necessary, please contact the sender for secure delivery instructions.

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Re: How to provide full text search over a union of one read-only and one mutable graph

[Brandon Sara <br...@collectivemedicaltech.com.INVALID>]

> A text index is "per dataset”.

This is what I figured, but wanted to be sure.

Thanks!
No PHI in Email: PointClickCare and Collective Medical, A PointClickCare Company, policies prohibit sending protected health information (PHI) by email, which may violate regulatory requirements. If sending PHI is necessary, please contact the sender for secure delivery instructions.

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Re: How to provide full text search over a union of one read-only and one mutable graph

[Andy Seaborne <an...@apache.org>]

On 01/10/2021 17:09, Brandon Sara wrote:
> Here is my scenario:
> - I have a large read-only set of data and a very small mutable set of data.
> - The read-only set of data can be pre-populated into a TDB2 while the mutable set of data will start out empty but will, obviously have data injected overtime during runtime.
> - I want one Fuseki service which combines both of these sets of data into a single dataset so that inference can be performed for the mutable data dependent upon the read-only data. (I.E. union them as named graphs into a single dataset).
> - I know that I can accomplish the union of the two sets of data as separate named graphs in the final dataset, but I also need full text search on both the read-only and the mutable portions of the final dataset.
> 
> How do I accomplish the need for indexing both read-only and mutable data that get combined into a single dataset? Do I create two indexes? Am I able to share indexes? I know that I could create an initial index from the read-only data which could then be updated when changes are made to the mutable data. However, once the read-only data needs updates (which it will from time to time), I need to regenerate the entire index in order to get any changes that weren’t added via an update through Fuseki (which is how the read-only data would be updated, likely a new tdb2 would be generated).

A text index is "per dataset".

A stack like:

* text dataset
* general dataset
   Setup models as needed
* storage

Rebuild the Lucence text index (this can be done offline outside Fuseki) 
when the read-only data changes then index the mutable data.

     Andy

> Thanks for the help!