You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Brian McBride <br...@epimorphics.com> on 2019/09/01 12:17:57 UTC
JenaText: support for explicit field names in text queries
The topic of this email was first mentioned in the recent thread about
JENA-1620 and Query Timeouts but is really a separate topic and
potentially gets a little complicated.
It used to be the case that JenaText supported querying of a Lucene text
index where the index was created independently of Jena and then made
available to JenaText via the dataset configuration. Is this still the
case?
Up until Jena 3.9.0 definitely, and I suspect 3.12.0 - I have not
confirmed this yet, it was possible to express text queries with field
names and they worked.
We have a Fuseki system in production (5+ years) that has "its own
mechanism"* for building a multi-field lucene index that is then queried
using JenaText. Those queries specify lucene field names as in the
example I gave in the earler thread:
[[
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX text: <http://jena.apache.org/text#>
PREFIX ppd: <http://landregistry.data.gov.uk/def/ppi/>
PREFIX lrcommon: <http://landregistry.data.gov.uk/def/common/>
SELECT * {
?ppd_propertyAddress
text:query ( "street: the" 3000000 ) .
} LIMIT 1
]]
You can try it on a system running Fuseki 3.9.0 here:
http://landregistry.data.gov.uk/app/qonsole
In a recent test with Jena 3.13.0-SNAPSHOT (from pull request #595)
installed in the dev version of that system, the query fails with a
query parse error. Do I need to do some extra configuration to get this
to work - e.g. specify a specific text query parser?
Rereading the Jena Text documentation I find:
[[
As mentioned earlier, the text index uses thenative Lucene query
language
<http://lucene.apache.org/core/6_4_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description>;
however, there are important constraints on how the Lucene query
language is used within jena-text. In particular,/explicit/references to
Lucene|Fields|with the|query string|*are not*supported. So how are
Lucene queries that would otherwise refer to multiple|Fields|expressed?
]]
The text goes on to explain the issues around the fact that JenaText
indexes each triple/quad as a separate document.
I have couched my question so far in terms of querying an externally
built text index, because that is the simplest, and possibly most
compelling way to ask the question and suggest not disallowing the use
of lucene fields in text queries. Not supported for creating indexes is
not the same as not supported for querying indexes.
I am (naively?) hoping that restoring the functionality to allow
specifying lucene field names in a text query is a quick fix for someone
familiar with the code. I am not familiar with the code, but am willing
to help where I can.
In the interests of full disclosure however, I should say that the
reason we have our own mechanism for building the text index is exactly
the one given in the JenaText documentation. We needed an index where
multiple properties of the same resource were indexed as a single
document. I would be happy to discuss this further - why the solution
indicated in the JenaText documentation didn't work for us and whether
there is way to construct a general purpose JenaText solution that
would. But there is a lot of potential for complexity there - and the
gears for a new Jena release are beginning to turn and I have been
hoping to deploy this new release when it becomes available.
Brian
* in fact we use JenaText with a custom TextDocProducer implementation.
--
------------------------------------------------------------------------
Brian McBride
brian.mcbride@epimorphics.com
Epimorphics Ltd www.epimorphics.com
Court Lodge, 105 High Street, Portishead, Bristol BS20 6PT
Tel: 01275 399069
Epimorphics Ltd. is a limited company registered in England (number 7016688)
Registered address: Court Lodge, 105 High Street, Portishead, Bristol
BS20 6PT, UK
Re: JenaText: support for explicit field names in text queries
Posted by Brian McBride <br...@epimorphics.com>.
Hi Chris,
Thanks for looking into this so promptly.
On 01/09/2019 21:12, Chris Tomlinson wrote:
> Hi again Brian,
>
> I looked a bit more and it’s not clear how to “fix” the issue after all. The change I suggested to TextIndexLucene uncovers a basic issue.
Such is life.
> When using a query such as:
>
> (?s ?score ?lit) text:query ( “some query string” 3000000 ) .
>
> The code currently inserts the primaryField, e.g., rdfs:label or what have you and then TextQueryPF binds the hit value from Lucene to the ?lit by looking up the matching field value in the result doc returned by Lucene; however, the change I suggested no longer defaults to the primaryField and so there’s an error during the result binding handling in TextQueryPF.
Yes - I tried running the unit tests and they barf. Most of them stop
barfing if the bind to ?var is disabled if it would try to bind null.
>
> The basic problem is that there’s an ambiguity with:
>
> … text:query ( “some query string” 3000000 ) .
>
> The current code doesn’t know whether there are fields mentioned in the query string or not.
>
> If there are fields in the query string then the use of the
>
> (?s ?score ?lit) text:query …
>
> form must be disallowed since there’s no way to know what field value to retrieve from the Lucene query result documents without further analysis of the query string.
Just so. If you try it in 3.9.0 (and presumably 3.12.0) a rather nasty
error occurs. So this is not new. We don't get the error because we
don't do that sort of query.
I have not understood the details of how the code finds the field
value. But if it could be arranged that if the document had just one
field - that was the one returned and passed to the PF and if there is
more than one, then null is passed to the PF Then the PF could throw a
meaningful exception if the value it gets is null when it tries to bind
it. That effectively disallows this query form on documents with more
than one field and does not require any analysis of the query. I
mention this tentatively as a possible approach - I don't feel I have a
strong grip on the code. It is definitely ugly but might be a possible
holding position whilst something better is worked out.
> Apparently in your application there will generally be two or more matching fields in each result document
yes
> and it would be further complicated to figure out what matching field value to use - or invent another syntax from grabbing more than a single ?lit per result doc.
>
> If there are no fields mentioned in the query string then the primaryField should be used explicitly and then ?lit can be bound to an appropriate match value as currently.
>
> Perhaps you can raise a Jena issue and we can discuss and see what can be done.
Done - https://jira.apache.org/jira/browse/JENA-1749
I'm conscious that there is a release now in progress and you are not
going be available to look at this. Do you have any thoughts on whether
3.13.0 should go ahead without at least an interim resolution to this issue?
Regards
Brian
>
> Regards,
> Chris
>
>
>
>> On Sep 1, 2019, at 2:25 PM, Chris Tomlinson <ch...@gmail.com> wrote:
>>
>> Hi Brian,
>>
>>> On Sep 1, 2019, at 7:17 AM, Brian McBride <brian.mcbride@epimorphics.com <ma...@epimorphics.com>> wrote:
>>>
>>> It used to be the case that JenaText supported querying of a Lucene text index where the index was created independently of Jena and then made available to JenaText via the dataset configuration. Is this still the case?
>> That should still be the case, with the proviso that currently the fields names be handled via RDF properties outside the query string.
>>
>> As you noted, it has been documented since 3.6.0 <https://jena.apache.org/documentation/query/text-query.html> that:
>>
>>> No explicit use of Fields within the query string is supported.
>> This is based on the assumption that the indexes contain only a single property field in the documents as they are indexed and hence only a single field corresponding to an RDF property in a query. Evidently a poor assumption not caught until now.
>>
>>
>>> Up until Jena 3.9.0 definitely, and I suspect 3.12.0 - I have not confirmed this yet, it was possible to express text queries with field names and they worked.
>> You’re correct, the change was introduced <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L744> in the 3.13.0 code that breaks the previous behavior. I’m not able to explore fixing this for the next three weeks but may take a look at “fixing” this then. The basic change would be to replace the referenced line by:
>>
>> qstring = qs;
>>
>> and that should be it. The results handling ( in simpleResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L562> and highlightResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L668>) should need no changes since Lucene:
>>
>> doc.get(null)
>>
>> just returns null which is already handled. Evidently your application doesn’t use the
>>
>> (?s ?score ?lit) text:query …
>>
>> form, since there’s no information about what fields have been used in the queryString no bindings for ?lit can be made.
>>
>>> We needed an index where multiple properties of the same resource were indexed as a single document. I would be happy to discuss this further - why the solution indicated in the JenaText documentation didn't work for us and whether there is way to construct a general purpose JenaText solution that would.
>>
>> More explanation would be interesting.
>>
>> Sorry for the inconvenience,
>> Chris
>>
>
--
------------------------------------------------------------------------
Brian McBride
brian.mcbride@epimorphics.com
Epimorphics Ltd www.epimorphics.com
Court Lodge, 105 High Street, Portishead, Bristol BS20 6PT
Tel: 01275 399069
Epimorphics Ltd. is a limited company registered in England (number 7016688)
Registered address: Court Lodge, 105 High Street, Portishead, Bristol
BS20 6PT, UK
Re: JenaText: support for explicit field names in text queries
Posted by Chris Tomlinson <ch...@gmail.com>.
Hi again Brian,
I looked a bit more and it’s not clear how to “fix” the issue after all. The change I suggested to TextIndexLucene uncovers a basic issue.
When using a query such as:
(?s ?score ?lit) text:query ( “some query string” 3000000 ) .
The code currently inserts the primaryField, e.g., rdfs:label or what have you and then TextQueryPF binds the hit value from Lucene to the ?lit by looking up the matching field value in the result doc returned by Lucene; however, the change I suggested no longer defaults to the primaryField and so there’s an error during the result binding handling in TextQueryPF.
The basic problem is that there’s an ambiguity with:
… text:query ( “some query string” 3000000 ) .
The current code doesn’t know whether there are fields mentioned in the query string or not.
If there are fields in the query string then the use of the
(?s ?score ?lit) text:query …
form must be disallowed since there’s no way to know what field value to retrieve from the Lucene query result documents without further analysis of the query string. Apparently in your application there will generally be two or more matching fields in each result document and it would be further complicated to figure out what matching field value to use - or invent another syntax from grabbing more than a single ?lit per result doc.
If there are no fields mentioned in the query string then the primaryField should be used explicitly and then ?lit can be bound to an appropriate match value as currently.
Perhaps you can raise a Jena issue and we can discuss and see what can be done.
Regards,
Chris
> On Sep 1, 2019, at 2:25 PM, Chris Tomlinson <ch...@gmail.com> wrote:
>
> Hi Brian,
>
>> On Sep 1, 2019, at 7:17 AM, Brian McBride <brian.mcbride@epimorphics.com <ma...@epimorphics.com>> wrote:
>>
>> It used to be the case that JenaText supported querying of a Lucene text index where the index was created independently of Jena and then made available to JenaText via the dataset configuration. Is this still the case?
>
> That should still be the case, with the proviso that currently the fields names be handled via RDF properties outside the query string.
>
> As you noted, it has been documented since 3.6.0 <https://jena.apache.org/documentation/query/text-query.html> that:
>
>> No explicit use of Fields within the query string is supported.
>
> This is based on the assumption that the indexes contain only a single property field in the documents as they are indexed and hence only a single field corresponding to an RDF property in a query. Evidently a poor assumption not caught until now.
>
>
>> Up until Jena 3.9.0 definitely, and I suspect 3.12.0 - I have not confirmed this yet, it was possible to express text queries with field names and they worked.
>
> You’re correct, the change was introduced <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L744> in the 3.13.0 code that breaks the previous behavior. I’m not able to explore fixing this for the next three weeks but may take a look at “fixing” this then. The basic change would be to replace the referenced line by:
>
> qstring = qs;
>
> and that should be it. The results handling ( in simpleResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L562> and highlightResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L668>) should need no changes since Lucene:
>
> doc.get(null)
>
> just returns null which is already handled. Evidently your application doesn’t use the
>
> (?s ?score ?lit) text:query …
>
> form, since there’s no information about what fields have been used in the queryString no bindings for ?lit can be made.
>
>> We needed an index where multiple properties of the same resource were indexed as a single document. I would be happy to discuss this further - why the solution indicated in the JenaText documentation didn't work for us and whether there is way to construct a general purpose JenaText solution that would.
>
>
> More explanation would be interesting.
>
> Sorry for the inconvenience,
> Chris
>
Re: JenaText: support for explicit field names in text queries
Posted by Chris Tomlinson <ch...@gmail.com>.
Hi Brian,
> On Sep 1, 2019, at 7:17 AM, Brian McBride <br...@epimorphics.com> wrote:
>
> It used to be the case that JenaText supported querying of a Lucene text index where the index was created independently of Jena and then made available to JenaText via the dataset configuration. Is this still the case?
That should still be the case, with the proviso that currently the fields names be handled via RDF properties outside the query string.
As you noted, it has been documented since 3.6.0 <https://jena.apache.org/documentation/query/text-query.html> that:
> No explicit use of Fields within the query string is supported.
This is based on the assumption that the indexes contain only a single property field in the documents as they are indexed and hence only a single field corresponding to an RDF property in a query. Evidently a poor assumption not caught until now.
> Up until Jena 3.9.0 definitely, and I suspect 3.12.0 - I have not confirmed this yet, it was possible to express text queries with field names and they worked.
You’re correct, the change was introduced <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L744> in the 3.13.0 code that breaks the previous behavior. I’m not able to explore fixing this for the next three weeks but may take a look at “fixing” this then. The basic change would be to replace the referenced line by:
qstring = qs;
and that should be it. The results handling ( in simpleResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L562> and highlightResults <https://github.com/apache/jena/blob/519c129ab2dfcb5eb43f1a337c618a8e69f88acd/jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java#L668>) should need no changes since Lucene:
doc.get(null)
just returns null which is already handled. Evidently your application doesn’t use the
(?s ?score ?lit) text:query …
form, since there’s no information about what fields have been used in the queryString no bindings for ?lit can be made.
> We needed an index where multiple properties of the same resource were indexed as a single document. I would be happy to discuss this further - why the solution indicated in the JenaText documentation didn't work for us and whether there is way to construct a general purpose JenaText solution that would.
More explanation would be interesting.
Sorry for the inconvenience,
Chris