You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Rupert Westenthaler <ru...@gmail.com> on 2012/10/13 10:23:51 UTC

Stanbol Semantic Indexing (was Re: Next releases)

Hi all,

This mail tries to provide an Overview about the Stanbol Semantic
Indexing module that covers the generalized parts of the architecture
described by STANBOL-471 and con be found in the
"contenthub-two-layered-structure" in the
"org.apache.stanbol.commons.semanticindex" module.

The remaining part of this Mail do use Markdown Syntax to allow later
usage on the Stanbol webpage.

@Suat: I think we should create a new Jira Issue about the Stanbol
Semantic Indexing module that supercedes STANBOL-498/499 and add the
information of this mail as description. WDYT?

## Semantic Indexing API

Development of the Stanbol Semantic Indexing API initialized by
[STANBOL-471](https://issues.apache.org/jira/browse/STANBOL-471).
[STANBOL-498](https://issues.apache.org/jira/browse/STANBOL-498) and
[STANBOL-499](https://issues.apache.org/jira/browse/STANBOL-499)
describe the main Interfaces and finally
[STANBOL-701](https://issues.apache.org/jira/browse/STANBOL-701)
describes the generalization of the first Contenthub specific
interfaces to the  "org.apache.stanbol.commons.semanticindex" module
that will also be adopted by a future version of the Entityhub.

**IndexingSource** and **Store** do represent the first (storage)
layer while **SemanticIndex** represents the second (indexing) layer
of the two-layerd design. In addition there are Managers that ease
OSGI services lookup for both layers. The following subsection do
describe the Interfaces in more detail.

### IndexingSource<Item>

An IndexingSource needs to support dereferencing functionality for
Items based on their ID

    :::java
    getItem(String id) : Item

method. It is intended to be used for integrating external services
with Apache Stanbol.

In addition it can inform about changes of the data

    :::java
    getEpoch() : long
    getRevision() : long
    changes(long epoch, long revision, int batchSize) : ChangeSet<Item>

If the *epoch* changes it indicates that the data provided by this
IndexingSource do have completely changed (e.g. useful if one loads a
new DBpedia Index). The *revision* indicates the state of the
IndexingSource at a given time. The idea is to use
*System#currentTimeMillis* for revisions. Note that multiple Items may
be changes within the same *revision*.

The most important vehicle is the ChangeSet<Item>. It is obtained by
asking the Store for all changes since a *revision* in the context of
an *epoch*. This request will be answered with a ChangeSet of changed
Items or an EpochException if the *epoch* changed. The ChangeSet will
include a maximum *batchSize* items (but at least all Items that do
use the same *revision* number).

Note that ChangeSets do NOT include the changed Items itself, but only
Ids. Consumers of the ChangeSet need to use the dereference
functionality of the Store to acquire the Item data. The reason for
that is to first reduce the size of ChangeSets and secondly it allows
also to separate the implementation of the Store with the
implementation of the revision management.

### Store<Item>

The Store represents an IndexingSource<Item> that supports full CRUD

    :::java
   remove(String uri) : Item
   put(Item item) : String
   put(Iterable<Item> items) : Iterable<String>
   remove(Iterable<String> uris) : void
   removeAll()


### SemanticIndex<Item>

This interface describes the second (indexing) layer. Components
implementing this interface need to cover both the indexing process
and the storage of the indexed data. The following three methods are
used for semantic indexing

    :::java
   index(Item item) : boolean
   remove(String uri) : void
   persist(long revision) : void
   getRevision : long
   getState() : IndexState

Assuming that a SemanticIndex<Item> is connected with a
IndexingSource<Items> than those methods will be called based on
changes provided by the IndexingSource via a ChangeSet.
First the #getRevision method is used to obtain the current state of
the SemanticIndex. Than a ChaneSet is obtained from the IndexingSource
by using this revision. Based on the ChangeSet Items are retrieved
from the IndexingSource. If the IndexingSource returns an Item than
index( is called. If the IndexingSource returns NULL, than remove is
called on the SemanticIndex.

Note that the naming of the persist(..) method is misleading as it is
not intended to persist changes but rather to update the *revision* of
the SemanticIndex.

**@Suat** maybe we should remove that method and parse the *revision*
as 2nd parameter to the index and remove method.

SemanticIndexes do have states: UNINT, INDEXING are used during the
initial indexing state. ACTIVE means that the index is in normal
operation and finally REINDEXING is used after an *epoch* change of
the IndexingSource. In this state the SemanticIndex can still be used
(with the data before the epoch change) while the re-indexing based on
the new data is preformed.

In the first version the Stanbol semanticindexing will not include a
component that provides an implementation of the above workflow, but I
think we will add such a component. If that is the case than a
SemanticIndex would only need to specify its IndexingSource(s) and
Stanbol would keep the index in sync with its source.

### Provided Services

The services API of the semanticindexing module does NOT include the
actual Java APIs for SemanticSearch but rather leave it to the
implementations to register those APIs themselves as OSGI services.
Stanbol already defines/uses a lot of those interfaces.
Implementations that implement those will naturally integrate To give
some Examples: A SemanticIndex storing its data in Solr can register
its SolrCore as OSGI service as described by [2]. SemanticIndexes
using a Clerezza TripleStore can be accessed via the Clerezza
TCManager and can expose a SPARQL endpoint as described in [3].

This design has the advantages that

* the semanticindexing API keeps focused on the semantic indexing
process and therefore easier to implement
* it allows greater flexibility and extensibility (e.g. one could
write a semantic index based on couchDB and register the RESTful and
Java API similar as it is done for Solr
* it allows both the storage as the semanticindex layer to provide
additional services (e.g. if a TripleStore is used to store the data
it can directly provide the SPARQL endpoint in case data are stored in
a CMS the SPARQL endpoint can be provides by a SemanticIndex that
knows how to convert the CMS data to RDF).
* it fits very well to the service oriented architecture of OSGI

BTW we will also use the same system for the Stanbol specific services
(e.g. the featured search of the Contenthub, the LDpath Backend
functionality or the FieldQuery service of the Entityhub.

### Next Steps:

The first Stanbol Component that will use this infrastructure will be
the Contenthub. Suat is the Person in charge for this. Based on the
version re-integrated with the trunk I will than continue the
development of the Entityhub on top of the semanticindexing module.

best
Rupert Westenthaler

[1] http://stanbol.apache.org/presentations/Stanbol_Overview_2012-04.pdf
[2] http://stanbol.apache.org/docs/trunk/utils/commons-solr
[3] http://markmail.org/message/zm2tqlvs4flwvjyd


-- 
| Rupert Westenthaler             rupert.westenthaler@gmail.com
| Bodenlehenstraße 11                             ++43-699-11108907
| A-5500 Bischofshofen

Re: Stanbol Semantic Indexing (was Re: Next releases)

Posted by Rupert Westenthaler <ru...@gmail.com>.
>>
>> SemanticIndexes do have states: UNINT, INDEXING are used during the
>> initial indexing state. ACTIVE means that the index is in normal
>> operation and finally REINDEXING is used after an *epoch* change of
>> the IndexingSource. In this state the SemanticIndex can still be used
>> (with the data before the epoch change) while the re-indexing based on
>> the new data is preformed.
>>
>> In the first version the Stanbol semanticindexing will not include a
>> component that provides an implementation of the above workflow, but I
>> think we will add such a component. If that is the case than a
>> SemanticIndex would only need to specify its IndexingSource(s) and
>> Stanbol would keep the index in sync with its source.
>
> In the first step Contenthub will provide an implementation the above
> workflow you mention with a Store (e.g FileStore[1] and SemanticIndex
> implementation[2] which is synchronized with the Store. Do you mean
> another (more generic) implementation?
>

Year. I think this workflow should be a service provided by Stanbol.
Maybe you can even start such a component when you implement the
ClerezzaIndex (needed to keep the SPARQL endpoint feature over the
enhancement metadata).

best
Rupert

-- 
| Rupert Westenthaler             rupert.westenthaler@gmail.com
| Bodenlehenstraße 11                             ++43-699-11108907
| A-5500 Bischofshofen

Re: Stanbol Semantic Indexing (was Re: Next releases)

Posted by Suat Gonul <su...@gmail.com>.
Hi Rupert,

On 10/13/2012 11:23 AM, Rupert Westenthaler wrote:
> Hi all,
>
> This mail tries to provide an Overview about the Stanbol Semantic
> Indexing module that covers the generalized parts of the architecture
> described by STANBOL-471 and con be found in the
> "contenthub-two-layered-structure" in the
> "org.apache.stanbol.commons.semanticindex" module.
>
> The remaining part of this Mail do use Markdown Syntax to allow later
> usage on the Stanbol webpage.
>
> @Suat: I think we should create a new Jira Issue about the Stanbol
> Semantic Indexing module that supercedes STANBOL-498/499 and add the
> information of this mail as description. WDYT?

I agree. STANBOL-471 is more about Contenthub component. Since we
introduce a common API on top of Contenthub, we can open a dedicate
issue for this.

>
> ## Semantic Indexing API
>
> Development of the Stanbol Semantic Indexing API initialized by
> [STANBOL-471](https://issues.apache.org/jira/browse/STANBOL-471).
> [STANBOL-498](https://issues.apache.org/jira/browse/STANBOL-498) and
> [STANBOL-499](https://issues.apache.org/jira/browse/STANBOL-499)
> describe the main Interfaces and finally
> [STANBOL-701](https://issues.apache.org/jira/browse/STANBOL-701)
> describes the generalization of the first Contenthub specific
> interfaces to the  "org.apache.stanbol.commons.semanticindex" module
> that will also be adopted by a future version of the Entityhub.
>
> **IndexingSource** and **Store** do represent the first (storage)
> layer while **SemanticIndex** represents the second (indexing) layer
> of the two-layerd design. In addition there are Managers that ease
> OSGI services lookup for both layers. The following subsection do
> describe the Interfaces in more detail.
>
> ### IndexingSource<Item>
>
> An IndexingSource needs to support dereferencing functionality for
> Items based on their ID
>
>     :::java
>     getItem(String id) : Item
>
> method. It is intended to be used for integrating external services
> with Apache Stanbol.
>
> In addition it can inform about changes of the data
>
>     :::java
>     getEpoch() : long
>     getRevision() : long
>     changes(long epoch, long revision, int batchSize) : ChangeSet<Item>
>
> If the *epoch* changes it indicates that the data provided by this
> IndexingSource do have completely changed (e.g. useful if one loads a
> new DBpedia Index). The *revision* indicates the state of the
> IndexingSource at a given time. The idea is to use
> *System#currentTimeMillis* for revisions. Note that multiple Items may
> be changes within the same *revision*.
>
> The most important vehicle is the ChangeSet<Item>. It is obtained by
> asking the Store for all changes since a *revision* in the context of
> an *epoch*. This request will be answered with a ChangeSet of changed
> Items or an EpochException if the *epoch* changed. The ChangeSet will
> include a maximum *batchSize* items (but at least all Items that do
> use the same *revision* number).
>
> Note that ChangeSets do NOT include the changed Items itself, but only
> Ids. Consumers of the ChangeSet need to use the dereference
> functionality of the Store to acquire the Item data. The reason for
> that is to first reduce the size of ChangeSets and secondly it allows
> also to separate the implementation of the Store with the
> implementation of the revision management.
>
> ### Store<Item>
>
> The Store represents an IndexingSource<Item> that supports full CRUD
>
>     :::java
>    remove(String uri) : Item
>    put(Item item) : String
>    put(Iterable<Item> items) : Iterable<String>
>    remove(Iterable<String> uris) : void
>    removeAll()
>
>
> ### SemanticIndex<Item>
>
> This interface describes the second (indexing) layer. Components
> implementing this interface need to cover both the indexing process
> and the storage of the indexed data. The following three methods are
> used for semantic indexing
>
>     :::java
>    index(Item item) : boolean
>    remove(String uri) : void
>    persist(long revision) : void
>    getRevision : long
>    getState() : IndexState
>
> Assuming that a SemanticIndex<Item> is connected with a
> IndexingSource<Items> than those methods will be called based on
> changes provided by the IndexingSource via a ChangeSet.
> First the #getRevision method is used to obtain the current state of
> the SemanticIndex. Than a ChaneSet is obtained from the IndexingSource
> by using this revision. Based on the ChangeSet Items are retrieved
> from the IndexingSource. If the IndexingSource returns an Item than
> index( is called. If the IndexingSource returns NULL, than remove is
> called on the SemanticIndex.
>
> Note that the naming of the persist(..) method is misleading as it is
> not intended to persist changes but rather to update the *revision* of
> the SemanticIndex.
>
> **@Suat** maybe we should remove that method and parse the *revision*
> as 2nd parameter to the index and remove method.

I'm OK with that.

>
> SemanticIndexes do have states: UNINT, INDEXING are used during the
> initial indexing state. ACTIVE means that the index is in normal
> operation and finally REINDEXING is used after an *epoch* change of
> the IndexingSource. In this state the SemanticIndex can still be used
> (with the data before the epoch change) while the re-indexing based on
> the new data is preformed.
>
> In the first version the Stanbol semanticindexing will not include a
> component that provides an implementation of the above workflow, but I
> think we will add such a component. If that is the case than a
> SemanticIndex would only need to specify its IndexingSource(s) and
> Stanbol would keep the index in sync with its source.

In the first step Contenthub will provide an implementation the above
workflow you mention with a Store (e.g FileStore[1] and SemanticIndex
implementation[2] which is synchronized with the Store. Do you mean
another (more generic) implementation?

Best,
Suat

[1]
https://svn.apache.org/repos/asf/stanbol/branches/contenthub-two-layered-structure/contenthub/store/file/src/main/java/org/apache/stanbol/contenthub/store/file/FileStore.java
<https://svn.apache.org/repos/asf/stanbol/branches/contenthub-two-layered-structure/contenthub/store/file/src/main/java/org/apache/stanbol/contenthub/store/file/FileStore.java>
[2]
https://svn.apache.org/repos/asf/stanbol/branches/contenthub-two-layered-structure/contenthub/index/src/main/java/org/apache/stanbol/contenthub/index/solr/SolrSemanticIndex.java
<https://svn.apache.org/repos/asf/stanbol/branches/contenthub-two-layered-structure/contenthub/index/src/main/java/org/apache/stanbol/contenthub/index/solr/SolrSemanticIndex.java>


>
> ### Provided Services
>
> The services API of the semanticindexing module does NOT include the
> actual Java APIs for SemanticSearch but rather leave it to the
> implementations to register those APIs themselves as OSGI services.
> Stanbol already defines/uses a lot of those interfaces.
> Implementations that implement those will naturally integrate To give
> some Examples: A SemanticIndex storing its data in Solr can register
> its SolrCore as OSGI service as described by [2]. SemanticIndexes
> using a Clerezza TripleStore can be accessed via the Clerezza
> TCManager and can expose a SPARQL endpoint as described in [3].
>
> This design has the advantages that
>
> * the semanticindexing API keeps focused on the semantic indexing
> process and therefore easier to implement
> * it allows greater flexibility and extensibility (e.g. one could
> write a semantic index based on couchDB and register the RESTful and
> Java API similar as it is done for Solr
> * it allows both the storage as the semanticindex layer to provide
> additional services (e.g. if a TripleStore is used to store the data
> it can directly provide the SPARQL endpoint in case data are stored in
> a CMS the SPARQL endpoint can be provides by a SemanticIndex that
> knows how to convert the CMS data to RDF).
> * it fits very well to the service oriented architecture of OSGI
>
> BTW we will also use the same system for the Stanbol specific services
> (e.g. the featured search of the Contenthub, the LDpath Backend
> functionality or the FieldQuery service of the Entityhub.
>
> ### Next Steps:
>
> The first Stanbol Component that will use this infrastructure will be
> the Contenthub. Suat is the Person in charge for this. Based on the
> version re-integrated with the trunk I will than continue the
> development of the Entityhub on top of the semanticindexing module.
>
> best
> Rupert Westenthaler
>
> [1] http://stanbol.apache.org/presentations/Stanbol_Overview_2012-04.pdf
> [2] http://stanbol.apache.org/docs/trunk/utils/commons-solr
> [3] http://markmail.org/message/zm2tqlvs4flwvjyd
>
>