[jira] [Commented] (STANBOL-1125) Create a lightweight EntityHub Indexing Tool for Freebase

["A. Soroka (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STANBOL-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018093#comment-14018093 ] 

A. Soroka commented on STANBOL-1125:
------------------------------------

Freebase is not the only source of data that would benefit from this kind of tool. I've been indexing dumps of the U.S. Library of Congress' Linked Data ([#note1]) and found the resource requirements onerous, and I'm going to have to set up some kind of special workflow to index the Virtual International Authority File. ([#note2]) Many of the data sources in which I am interested require little or no processing of the kind we would write in LDPath programs-- they are to be indexed virtually "plain". For those data sources that do require some processing of simple kinds (e.g. translating predicates), it might be possible to allow that as a step on a single Representation before storing that Representation into a Yard, instead of as a transduction over the whole store of RDF.

I would be interested in working on this problem. It seems to me at a first glance that with suitable restrictions on the inputs (perhaps N-Triples files sorted by subject URI?) a very performant streaming solution could be developed with a minimal cost in space for computation.

({anchor:note1}1) Controlled vocabularies for subject headings and name authorities widely used in library metadata, available at:

http://id.loc.gov/download/

({anchor:note2}2) Also a name authority system, but federated from the linked data of several different national libraries, available at:

http://viaf.org/viaf/data/

> Create a lightweight EntityHub Indexing Tool for Freebase
> ---------------------------------------------------------
>
>                 Key: STANBOL-1125
>                 URL: https://issues.apache.org/jira/browse/STANBOL-1125
>             Project: Stanbol
>          Issue Type: Improvement
>          Components: Entityhub
>            Reporter: Rafa Haro
>
> Due to the enormous size of the dumps, current Freebase indexing tool in Stanbol can't barely work in machines without several gigas of RAM and/or SSD disks. JenaTDB importer has been identified as the bootle neck of the indexing process. To use an RDF database is mandatory in order to, for instance, use LDPath programs at indexing time.
> The idea is to develop a lightweight indexing tool that stream data from the dumps and push it directly to Solr. Despite losing some functionality, it is possible for any user to generate Freebase EntityHub indexes from any dump.



--
This message was sent by Atlassian JIRA
(v6.2#6252)