You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stanbol.apache.org by "Olivier Grisel (JIRA)" <ji...@apache.org> on 2011/09/22 17:14:26 UTC

[jira] [Created] (STANBOL-330) EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher

EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher
--------------------------------------------------------------------------------

                 Key: STANBOL-330
                 URL: https://issues.apache.org/jira/browse/STANBOL-330
             Project: Stanbol
          Issue Type: Bug
            Reporter: Olivier Grisel


At the moment the fulltext expression is tokenized on the client side and the bif:contains clause contains a conjunction of single word phrase queries which can be invalid if one of the word is a stop word.

Instead of tokenizing on the client size one should probably just do a basic fulltext search without explicit conjunction and phrase query operators.

Detailed error message to be added in the first comment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (STANBOL-330) EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher

Posted by "Rupert Westenthaler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/STANBOL-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rupert Westenthaler resolved STANBOL-330.
-----------------------------------------

    Resolution: Fixed
      Assignee: Rupert Westenthaler

Fixed in revision 1175921

> EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher
> --------------------------------------------------------------------------------
>
>                 Key: STANBOL-330
>                 URL: https://issues.apache.org/jira/browse/STANBOL-330
>             Project: Stanbol
>          Issue Type: Bug
>            Reporter: Olivier Grisel
>            Assignee: Rupert Westenthaler
>
> At the moment the fulltext expression is tokenized on the client side and the bif:contains clause contains a conjunction of single word phrase queries which can be invalid if one of the word is a stop word.
> Instead of tokenizing on the client size one should probably just do a basic fulltext search without explicit conjunction and phrase query operators.
> Detailed error message to be added in the first comment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (STANBOL-330) EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher

Posted by "Olivier Grisel (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/STANBOL-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114210#comment-13114210 ] 

Olivier Grisel commented on STANBOL-330:
----------------------------------------

Thanks for your analysis. My initial investigations were wrong. I agree with your suggested changes.

> EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher
> --------------------------------------------------------------------------------
>
>                 Key: STANBOL-330
>                 URL: https://issues.apache.org/jira/browse/STANBOL-330
>             Project: Stanbol
>          Issue Type: Bug
>            Reporter: Olivier Grisel
>
> At the moment the fulltext expression is tokenized on the client side and the bif:contains clause contains a conjunction of single word phrase queries which can be invalid if one of the word is a stop word.
> Instead of tokenizing on the client size one should probably just do a basic fulltext search without explicit conjunction and phrase query operators.
> Detailed error message to be added in the first comment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (STANBOL-330) EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher

Posted by "Olivier Grisel (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/STANBOL-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112642#comment-13112642 ] 

Olivier Grisel commented on STANBOL-330:
----------------------------------------

Querying  the TextConstraint expression with "Richard M. Daley" triggers the exception:

SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: org.apache.stanbol.entityhub.servicesapi.site.ReferencedSiteException: Unable execute Query on remote site http://dbpedia.org/sparql
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.executeQuery(ReferencedSiteRootResource.java:340)
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.queryEntities(ReferencedSiteRootResource.java:313)
	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1465)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1396)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1345)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1335)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:96)
	at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:79)
	at org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)
	at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)
	at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
	at org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)
	at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)
	at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.stanbol.entityhub.servicesapi.site.ReferencedSiteException: Unable execute Query on remote site http://dbpedia.org/sparql
	at org.apache.stanbol.entityhub.core.impl.ReferencedSiteImpl.find(ReferencedSiteImpl.java:389)
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.executeQuery(ReferencedSiteRootResource.java:335)
	... 42 more
Caused by: java.io.IOException: GET with Content: 
Virtuoso 37000 Error XM028: Free-text expression, line 1: phrase consists of noise words exclusively
in the following expression:
[ __enc "UTF-8" ] ("Richard" AND "M" AND "." AND "Daley")

SPARQL query:
define sql:big-data-const 0 define output:format "HTTP+TTL text/rdf+n3" 
#output-format:text/rdf+n3
CONSTRUCT { 
  ?id <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?v_1 .
  ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_2 .
  <http://www.iks-project.eu/ontology/rick/query/QueryResultSet> <http://www.iks-project.eu/ontology/rick/query/queryResult> ?id . 
} WHERE { 
  { 
    SELECT ?id 
    WHERE { 
      ?id <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Person> . 
      ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_2 . 
        ?v_2 bif:contains '("Richard" AND "M" AND "." AND "Daley")' . 
    } 
    ORDER BY DESC ( <LONG::IRI_RANK> (?id) ) 
    LIMIT 3 
        } 
  OPTIONAL { ?id <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?v_1 . } 
  OPTIONAL { ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_2 . } 
} 

	at org.apache.stanbol.entityhub.site.linkeddata.impl.SparqlEndpointUtils.sendSparqlRequest(SparqlEndpointUtils.java:65)
	at org.apache.stanbol.entityhub.site.linkeddata.impl.VirtuosoSearcher.find(VirtuosoSearcher.java:66)
	at org.apache.stanbol.entityhub.core.impl.ReferencedSiteImpl.find(ReferencedSiteImpl.java:387)
	... 43 more
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://dbpedia.org/sparql?query=CONSTRUCT+%7B+%0A++?id+%3Chttp://www.w3.org/1999/02/22-rdf-syntax-ns%23type%3E+?v_1+.%0A++?id+%3Chttp://www.w3.org/2000/01/rdf-schema%23label%3E+?v_2+.%0A++%3Chttp://www.iks-project.eu/ontology/rick/query/QueryResultSet%3E+%3Chttp://www.iks-project.eu/ontology/rick/query/queryResult%3E+?id+.+%0A%7D+WHERE+%7B+%0A++%7B+%0A++++SELECT+?id+%0A++++WHERE+%7B+%0A++++++?id+%3Chttp://www.w3.org/1999/02/22-rdf-syntax-ns%23type%3E+%3Chttp://dbpedia.org/ontology/Person%3E+.+%0A++++++?id+%3Chttp://www.w3.org/2000/01/rdf-schema%23label%3E+?v_2+.+%0A++++++++?v_2+bif:contains+'(%22Richard%22+AND+%22M%22+AND+%22.%22+AND+%22Daley%22)'+.+%0A++++%7D+%0A++++ORDER+BY+DESC+(+%3CLONG::IRI_RANK%3E+(?id)+)+%0A++++LIMIT+3+%0A++++++++%7D+%0A++OPTIONAL+%7B+?id+%3Chttp://www.w3.org/1999/02/22-rdf-syntax-ns%23type%3E+?v_1+.+%7D+%0A++OPTIONAL+%7B+?id+%3Chttp://www.w3.org/2000/01/rdf-schema%23label%3E+?v_2+.+%7D+%0A%7D+%0A&format=text/rdf%2Bn3
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1403)
	at org.apache.stanbol.entityhub.site.linkeddata.impl.SparqlEndpointUtils.sendSparqlRequest(SparqlEndpointUtils.java:53)
	... 45 more
Sep 22, 2011 5:06:10 PM com.sun.jersey.spi.container.ContainerResponse logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: org.apache.stanbol.entityhub.servicesapi.site.ReferencedSiteException: Unable execute Query on remote site http://dbpedia.org/sparql
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.executeQuery(ReferencedSiteRootResource.java:340)
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.queryEntities(ReferencedSiteRootResource.java:313)
	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1465)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1396)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1345)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1335)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:96)
	at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:79)
	at org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)
	at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)
	at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
	at org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)
	at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)
	at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.stanbol.entityhub.servicesapi.site.ReferencedSiteException: Unable execute Query on remote site http://dbpedia.org/sparql
	at org.apache.stanbol.entityhub.core.impl.ReferencedSiteImpl.find(ReferencedSiteImpl.java:389)
	at org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource.executeQuery(ReferencedSiteRootResource.java:335)
	... 42 more


> EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher
> --------------------------------------------------------------------------------
>
>                 Key: STANBOL-330
>                 URL: https://issues.apache.org/jira/browse/STANBOL-330
>             Project: Stanbol
>          Issue Type: Bug
>            Reporter: Olivier Grisel
>
> At the moment the fulltext expression is tokenized on the client side and the bif:contains clause contains a conjunction of single word phrase queries which can be invalid if one of the word is a stop word.
> Instead of tokenizing on the client size one should probably just do a basic fulltext search without explicit conjunction and phrase query operators.
> Detailed error message to be added in the first comment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

[jira] [Commented] (STANBOL-330) EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher

Posted by "Rupert Westenthaler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/STANBOL-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113249#comment-13113249 ] 

Rupert Westenthaler commented on STANBOL-330:
---------------------------------------------

In general the tokenizing of the search is done to provide a similar behavior to full text searches in Solr/Lucene based on the capabilities of [1]. Without this Queries on Virtuoso servers would behave like Solr queries in untokenized fields.

To the specific Example:

If I make the request (*)

    curl -v -X POST -d "name=Richard M. Daley" http://localhost:8080/entityhub/site/dbpedia/find

I do get "http://dbpedia.org/resource/Richard_M._Daley" as result.

The SPQRQL query sent to the dbpedia Virtuoso servers looks like this:

    CONSTRUCT { 
      ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_1 .
      <http://www.iks-project.eu/ontology/rick/query/QueryResultSet> <http://www.iks-project.eu/ontology/rick/query/queryResult> ?id . 
    } WHERE { 
      { 
        SELECT ?id 
        WHERE { 
          ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_1 . 
            ?v_1 bif:contains '("Richard" AND "M." AND "Daley")' . 
        } 
        ORDER BY DESC ( <LONG::IRI_RANK> (?id) ) 
        LIMIT 10 
            } 
      OPTIONAL { ?id <http://www.w3.org/2000/01/rdf-schema#label> ?v_1 . } 
    } 
    ORDER BY DESC ( <LONG::IRI_RANK> (?id) ) 
    LIMIT 10 

Only if I modify the search to "name=Richard M . Daley" than I get the reported error. Also  "name=Richard M Daley" is fine but executes much slower (3.5sec instead of 1sec for the query using "M."). Also tests with number (such as "Richard Daley 5") executed successfully.

Interesting is that "name=Richard .M Daley" also selects "http://dbpedia.org/resource/Richard_M._Daley". Because of this I come to the conclusion that the used full text search engine simple ignores non Alpha-Numeric characters. This could also explain the error, because it would end up with an empty token.

The documentation [1] section "20.3.3. Text Expression Syntax" notes that "A word is a sequence of word characters." but does not explicitly define what word characters are. For now I think we should simple ignore Tokens that do not contain a single Alpha-Numeric character. 

Based on that I suggest to modify the Virtuoso searcher to ignore tokens that do not contain a single Alpha-Numeric chars. However I would not filter non Alpha-Numeric chars because it looks like they are anyway filtered by the Virtuoso.

If you agree I will make the necessary changes and mark this Issue as resolved.

best
Rupert

(*) For this tests I used the stable launcher of the current version. Stoped/Uninstalled the defaultdata bundle for dbpedia and installed the dbpediacached bundle (org.apache.stanbol.data.sites.dbpedia.cached).

[1] http://docs.openlinksw.com/virtuoso/queryingftcols.html#containspredicate

> EnityHub query API can generate incorrect fulltext clause with Virtuoso searcher
> --------------------------------------------------------------------------------
>
>                 Key: STANBOL-330
>                 URL: https://issues.apache.org/jira/browse/STANBOL-330
>             Project: Stanbol
>          Issue Type: Bug
>            Reporter: Olivier Grisel
>
> At the moment the fulltext expression is tokenized on the client side and the bif:contains clause contains a conjunction of single word phrase queries which can be invalid if one of the word is a stop word.
> Instead of tokenizing on the client size one should probably just do a basic fulltext search without explicit conjunction and phrase query operators.
> Detailed error message to be added in the first comment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira