Re: [Apache Bloodhound] Proposals/BEP-0004/ResourceQuery added

[Andrej Golcov <an...@digiverse.si>]

Hi all,

I moved search query requirements, query syntax and use cases into separate
page:
https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0004/ResourceQuery

It is just first draft, so let's discuss how new search query should look
like and what features we want from it.

Regards, Andrej

On 28 November 2012 16:17, Apache Bloodhound <
bloodhound-dev@incubator.apache.org> wrote:

> Page "Proposals/BEP-0004/ResourceQuery" was added by andrej
> Content:
> -------8<------8<------8<------8<------8<------8<------8<------8<--------
> = Resource Query component
> [[PageOutline]]
> == Introduction #introduction
>
> This page describes functionality of Resource Query component. Resource
> Query component is responsible for resource indexing and query execution.
> It is not responsible for representation of search results to user. For
> overview of search and query solution see [wiki:BEP-0004].
>
> Usually user will not access to Resource Query component directly but via
> UI frontends e.g. search page, widget or wiki macro. Consider below a
> simple search workflow:
>  1. User searches for “bla status:closed” string in quick search box
>  1. Quick search forwards user to search page with URL
> …?q=bla20%status:closed
>  1. Search page calls Resource Query component and calls Resource Query
> component with query ”bla status:closed” and other parameters e.d. fields,
> sort etc.
>  1. Search page renders query results in appropriate way
>
> Resource Query component will provide a !ResourceQuery.query method with
> the following parameters:
>  * '''query''': query string e.g. “bla status:closed” or a parsed
> representation of the query . For more information see [#query_syntax Query
> syntax].
>  * '''sort''': optional sorting
>  * '''boost''': optional list of fields with boost values e.g. {“id”:
> 1000, “subject” :100, “description”:10}. Used only for score based sorting.
>  * '''filters''': optional list of terms. Usually can be cached by
> underlying search framework. For example {“type”: “wiki”}
>  * '''fields''': list of fields to return
>  * optional paging fields: '''rows/start''' or '''page/pagesize''' fields
>  * '''facets''' - optional list of facet terms, can be field or expression.
>
> == Resource Query is not a report tool #notreport
> As it was discussed on dev mailing list, search and query serve a
> different purpose than reports. Resource Query is not intended not provide
> complex SQL like expressions linke JOIN, UNION etc. Resource Query will
> search through flattened resource representation. Query syntax should
> support issue tracker specifics such as search through attachments, related
> tickets etc.
>
> == Query Syntax #query_syntax
> Resource Query will accept [
> http://lucene.apache.org/core/old_versioned_docs/versions/2_9_1/queryparsersyntax.htmlLucene-like] syntax familiar to users of Solr, [
> http://packages.python.org/Whoosh/querylang.html Whoosh], Haystack, [
> http://code.google.com/p/unladen-swallow/issues/searchtips Google Code]
> and [
> http://confluence.jetbrains.com/display/YTD4/Search+and+Command+Attributes#SearchandCommandAttributes-GeneralSearchAttributesYouTrack] with additional functions/meta tags specific for Bloodhound/Trac
> e.g. related tickets, attachment etc.
>
> Default Resource Query operator is AND.
>
> Bloodhound should provide it’s own query parser in order to be independent
> from underlying search platforms.
>
> === Issue-tracker specifics #tracker_specifics
> Resource Query should be able to search through Bloodhound specific
> fields/functions:
>  * comments
>  * attachments
>  * history
>  * related resources with different relation types: linked, duplicated,
> blocked, child/parent etc.
>
> Resource Query should support version changing, similar to WAS and CHANGED
> operator in JIRA (
> https://confluence.atlassian.com/display/JIRA/Advanced+Searching#AdvancedSearching-WAS,
> http://confluence.jetbrains.com/display/YTD4/Search+and+Command+Attributes
> )
>
> Other functions or meta tags can be used in query. Meta tags can be marked
> with specific character e.g. “#” (similar to YouTrack special keywords -
> http://confluence.jetbrains.com/display/YTD4/Search+and+Command+Attributes#SearchandCommandAttributes-ShortcutKeywords
> ):
>  * #me - current user
>  * #my - assigned to me
>  * #currentProject
>  * #ticket, #wiki etc.
>  * date and time helper functions e.g. 2weeksago, 1yearago etc.
>
> Indexing and query syntax must be easily extended by plugins. Here is not
> a complete list of other possible meta tags that can be provided by
> additional plugins:
>  * #resolved/unresolved - status:(resolved OR closed)
>  * version aggregation e.g. earliestUnreleasedVersion
>  * #hasAttachment
>  * code:xxx ... - contains code in wiki format
>  * #duplicated
>  * #closed = status:closed
>  * #yesterday
>  * ...
>
> == Use cases #usecases
> === User uses free text search or query  in quick search box
> #usecase_freesearch
> User inputs text  or query string in search box. The input can be directly
> propagated to query parameter, for example:
>  * bla
>  * open issue
>  * bla “open issue”
>  * bla status:open
>  * status:open
>
> === Possibility to specify what fields to return #usecase_fields
> Search page or widget must specify fields parameter of ResourceQuery.query
>  method.
> {{{
> #!python
> resourceQuery.query(fields=("id" , "title", "status"),...)
> }}}
>
> === Boolean operators and grouping #usecase_grouping
> Resource Query must support AND, OR, NOT and grouping (default operator is
> AND). Query string may look like:
>  * alpha AND NOT (beta OR gamma)
>  * “render AND shading”  - expression is equal to “render shading”
>  * title:x OR ( title:y AND message:z)
>
> === User can search using range expression #usecase_range
> Query string parameter should support inclusive and exclusive range
> expression, for example:
>  * date:[20050101 TO 20090715]
>  * title:{Aida TO Carmen}
>  * [0025 TO]
>  * {TO suffix}
>
> === Facets support #usecase_facets
> Query must support facets  (e.g. Resources: Tickets(10), Wiki (20)),
> Status (Open (22), Closed(33)) etc. Facets parameter should be used for
> this purposes.
>
> {{{
> #!python
> resourceQuery.query(facets=("type", "status"), ...)
> }}}
>
> === Flexible sorting #usecase_sorting
> Default sort order of text-search should be based on score and change
> date. Search page can set the following parameters for !ResourceQuery.query
> method:
> {{{
> #!python
> resourceQuery.query( sort = {"score":ASC, "change_date": DESC},
>   boost = {"id" : 1000, "subject" : 100, "description": 10},...)
> }}}
>
> === Paging support #usecase_paging
> Search page will represent query results in pages. For this purposes, it
> should use the following parameters of !ResourceQuery.query method.
> {{{
> #!python
> resourceQuery.query(start = 100, rows=50, ...)
> }}}
> or
> {{{
> #!python
> resourceQuery.query(pagesize=50, page=3, ...)
> }}}
>
> === Related ticket use case #usecase_ralated
> User queries tickets related to tickets that were reopened in last 14
> days. The query can be exprese with the following call:
> {{{
> #!python
> resourceQuery.query(
>   query="changed.status_from:open changed_date:[1weekago TO]",
>   facets=("parent_ticket"), ...
> )
> }}}
>
> === Search in comments #usecase_comments
> {{{
> #!python
> resourceQuery.query(query="attachment:bla", ...)
> }}}
>
> === Show in attachment  #usecase_comments
> {{{
> #!python
> resourceQuery.query(query="attachment:bla", ...)
> }}}
>
> === Show tickets that were commented since yesterday.
>  #usecase_last_commented
> {{{
> #!python
> resourceQuery.query(query="last_commented:[yesterday TO]", ...)
> }}}
>
> === Show all resources in current project that links to a ticket
> #usecase_project_linked
> {{{
> #!python
> resourceQuery.query(query="#currentProject AND linked:#123", ...)
> }}}
> -------8<------8<------8<------8<------8<------8<------8<------8<--------
>
> --
> Page URL: <
> https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0004/ResourceQuery
> >
> Apache Bloodhound <https://issues.apache.org/bloodhound/>
> The Apache Bloodhound (incubating) issue tracker
>
> This is an automated message. Someone added your email address to be
> notified of changes on 'Proposals/BEP-0004/ResourceQuery' page.
> If it was not you, please report to .
>