You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@marmotta.apache.org by Prashant <pn...@icloud.com> on 2014/07/02 15:16:52 UTC

Marmotta as a linked data platform

Hi 

I am planning to use Marmotta linked data platform in production to build applications for my client. My main requirements are

1. Sparql Query speed like native memory storage like Jena TDB. 
2. Versioning so that user can get snapshot of data at specified date by sparql queries.
3. Cache (Good to have hazel cast)
4. Text Search
5. Internationalization


I was playing with binary of Apache Marmotta 3.2.1 and found numerous issues while doing administration. It appears that it has been published in May 2013. Since then there is no major release. So I moved to latest git source and I found it has lots of good integration with cache, different backend like titan etc.

My questions here are

1. How stable is current source code is? can it be used in production like setup.
2. Is there any documentation to build and release current source code?

All suggestion and advice are welcome.

Thanks Again
Prashant








Thanks
Prashant

Re: Marmotta as a linked data platform

Posted by Prashant <pn...@icloud.com>.

Thanks Again Sebastian
See my response inline.
Also attaching a output of first 17million triples load. Not sure you will get this image file as attachment.but in case you get peak performance was close to 14k per sec. My load was in RDF/XML. Other format will get much quicker load.


Thanks
Prashant





On Jul 7, 2014, at 11:51 AM, Sebastian Schaffert <se...@gmail.com> wrote:

> Hi,
> 
> the KiWi versioning still works for interactive access if you enable it afterwards, but currently not for the bulk loading you describe (could be implemented though if I find the time, but it will slow down the loading performance to about half the speed). From experience, PostgreSQL tuning has a dramatic impact on performance, but I cannot really say if it will match Jena TDB (which is using its own custom data storage format optimized for triples). Maximum bulk load performance was never the main goal when developing the KiWi triple store, so you might be better of with TDB in your scenario. Still, your performance stats sound pretty low. Even on a slow machine I get at least 10.000 triples/sec (but typically much higher), which would give you about half the time you describe. Did you add the option to drop indexes before import (-I)?

Yes I did add drop index option and I see load was quicker.
> 
> Regarding your versioning requirement: this really sounds like named graphs are enough in your case, since your versions are really big. The KiWi versioning model was designed rather with the typical small user interactions (e.g. SPARQL updates with both deletes and adds) in mind, not with long-running transactions.
thanks this was good insight for my decision making.
> 
> I don't have much experience with the other backends, I just implemented interfaces so it works in test cases. No bulk loads so far. But Titan is said to have very good performance when using a Cassandra database as backend. If you really want to play with frequent bulk loading it might be worth playing with it.
> 
Yes I was reading Titan non RDF based graph experiment a lot particularly their Pearson case study. Probably will check back later.

> 
> 
> 
> 2014-07-07 12:13 GMT+02:00 Prashant <pn...@icloud.com>:
> Thank you all, your reply & time is much appreciated. 
> 
> For me main attraction of Kiwi as a backend was versioning but bulk loader does not support versioning, so I need to look at custom versioning anyway. May be named graph is way to go but that is for another day. 
> 
> In my use case bulk data loading is going to be frequent so I need fast data loading and on top of that versioning.  That said after doing some data loading exercise with Jena TDB vs kiwi with close to 30 million triples (which is not much) I see Jena is clear winner. It just take 6-7 minute on a 8 G mac machine while Kiwi (backed by Postgres) takes close to 90 minute. I know postgres tuning like more shared buffer (current 128 mb),  can improve loading performance but I do not think it will match Jena TDB. Any thought?
> 
> My another question is where we stand with Titan or Berkleydb  as a backend. I have both frequent write and read requirement in my use case. Although read will anyway surpass write but it is not write once use case...
> 
> Thanks
> Prashant
> 
> 
> 
> 
> 
> On Jul 7, 2014, at 10:53 AM, Sebastian Schaffert <se...@gmail.com> wrote:
> 
>> Hi all, 
>> 
>> some additions to Sergio's reply:
>> 
>> 
>> 2014-07-07 8:11 GMT+02:00 Sergio Fernández <wi...@apache.org>:
>> 
>> 
>> My main requirements are
>> 
>> 1. Sparql Query speed like native memory storage like Jena TDB.
>> 
>> We do not have performance figure about the KiWi triple store in comparison win Jena TDB or other. If you get some, please share it with the community to have a reference.
>> 
>> Please, take into account that KiWi is just the default triple store. In Marmotta you can easily use any Sesame-based triples store. Further details at:
>> 
>> http://marmotta.apache.org/platform/backends
>> 
>> Right now, SPARQL is NOT very fast in KiWi (compared to Virtuoso, but still faster than in some other Sesame backend). We are still working on improving SPARQL performance, but since SPARQL is by its very nature a very expressive query language not everything can be super-fast.
>>  
>> 4. Text Search
>> 
>> That feature was not coming to Marmotta from LMF (http://lmf.googlecode.com). But you should be able to still use it by adding this dependency to your webapp launcher:
>> 
>>   <dependency>
>>     <groupId>at.newmedialab.lmf</groupId>
>>     <artifactId>lmf-search</artifactId>
>>     <version>3.2.0-SNAPSHOT</version>
>>   </dependency>
>> 
>> All details at http://code.google.com/p/lmf/wiki/ModuleSemanticSearch
>> 
>> This is not necessarily needed. The KiWi SPARQL implementation since last version contains features for full-text search as part of SPARQL queries when using PostgreSQL:
>> 
>> http://marmotta.apache.org/kiwi/sparql.html
>> 
>> 
>> Greetings,
>> 
>> Sebastian 
> 
>

Re: Marmotta as a linked data platform

Posted by Sebastian Schaffert <se...@gmail.com>.

Hi,

the KiWi versioning still works for interactive access if you enable it
afterwards, but currently not for the bulk loading you describe (could be
implemented though if I find the time, but it will slow down the loading
performance to about half the speed). From experience, PostgreSQL tuning
has a dramatic impact on performance, but I cannot really say if it will
match Jena TDB (which is using its own custom data storage format optimized
for triples). Maximum bulk load performance was never the main goal when
developing the KiWi triple store, so you might be better of with TDB in
your scenario. Still, your performance stats sound pretty low. Even on a
slow machine I get at least 10.000 triples/sec (but typically much higher),
which would give you about half the time you describe. Did you add the
option to drop indexes before import (-I)?

Regarding your versioning requirement: this really sounds like named graphs
are enough in your case, since your versions are really big. The KiWi
versioning model was designed rather with the typical small user
interactions (e.g. SPARQL updates with both deletes and adds) in mind, not
with long-running transactions.

I don't have much experience with the other backends, I just implemented
interfaces so it works in test cases. No bulk loads so far. But Titan is
said to have very good performance when using a Cassandra database as
backend. If you really want to play with frequent bulk loading it might be
worth playing with it.

Sebastian

2014-07-07 12:13 GMT+02:00 Prashant <pn...@icloud.com>:

> Thank you all, your reply & time is much appreciated.
>
> For me main attraction of Kiwi as a backend was versioning but bulk loader
> does not support versioning, so I need to look at custom versioning anyway.
> May be named graph is way to go but that is for another day.
>
> In my use case bulk data loading is going to be frequent so I need fast
> data loading and on top of that versioning.  That said after doing some
> data loading exercise with Jena TDB vs kiwi with close to 30 million
> triples (which is not much) I see Jena is clear winner. It just take 6-7
> minute on a 8 G mac machine while Kiwi (backed by Postgres) takes close to
> 90 minute. I know postgres tuning like more shared buffer (current 128 mb),
>  can improve loading performance but I do not think it will match Jena TDB.
> Any thought?
>
> My another question is where we stand with Titan or Berkleydb  as a
> backend. I have both frequent write and read requirement in my use case.
> Although read will anyway surpass write but it is not write once use case...
>
> Thanks
> Prashant
>
>
>
>
>
> On Jul 7, 2014, at 10:53 AM, Sebastian Schaffert <
> sebastian.schaffert@gmail.com> wrote:
>
> Hi all,
>
> some additions to Sergio's reply:
>
>
> 2014-07-07 8:11 GMT+02:00 Sergio Fernández <wi...@apache.org>:
>>
>>
>>
>>  My main requirements are
>>>
>>> 1. Sparql Query speed like native memory storage like Jena TDB.
>>>
>>
>> We do not have performance figure about the KiWi triple store in
>> comparison win Jena TDB or other. If you get some, please share it with the
>> community to have a reference.
>>
>> Please, take into account that KiWi is just the default triple store. In
>> Marmotta you can easily use any Sesame-based triples store. Further details
>> at:
>>
>> http://marmotta.apache.org/platform/backends
>
>
> Right now, SPARQL is NOT very fast in KiWi (compared to Virtuoso, but
> still faster than in some other Sesame backend). We are still working on
> improving SPARQL performance, but since SPARQL is by its very nature a very
> expressive query language not everything can be super-fast.
>
>
>>  4. Text Search
>>>
>>
>> That feature was not coming to Marmotta from LMF (
>> http://lmf.googlecode.com). But you should be able to still use it by
>> adding this dependency to your webapp launcher:
>>
>>   <dependency>
>>     <groupId>at.newmedialab.lmf</groupId>
>>     <artifactId>lmf-search</artifactId>
>>     <version>3.2.0-SNAPSHOT</version>
>>   </dependency>
>>
>> All details at http://code.google.com/p/lmf/wiki/ModuleSemanticSearch
>
>
> This is not necessarily needed. The KiWi SPARQL implementation since last
> version contains features for full-text search as part of SPARQL queries
> when using PostgreSQL:
>
> http://marmotta.apache.org/kiwi/sparql.html
>
>
> Greetings,
>
> Sebastian
>
>
>

Re: Marmotta as a linked data platform

Posted by Prashant <pn...@icloud.com>.

Thank you all, your reply & time is much appreciated. 

For me main attraction of Kiwi as a backend was versioning but bulk loader does not support versioning, so I need to look at custom versioning anyway. May be named graph is way to go but that is for another day. 

In my use case bulk data loading is going to be frequent so I need fast data loading and on top of that versioning.  That said after doing some data loading exercise with Jena TDB vs kiwi with close to 30 million triples (which is not much) I see Jena is clear winner. It just take 6-7 minute on a 8 G mac machine while Kiwi (backed by Postgres) takes close to 90 minute. I know postgres tuning like more shared buffer (current 128 mb),  can improve loading performance but I do not think it will match Jena TDB. Any thought?

My another question is where we stand with Titan or Berkleydb  as a backend. I have both frequent write and read requirement in my use case. Although read will anyway surpass write but it is not write once use case...

Thanks
Prashant

On Jul 7, 2014, at 10:53 AM, Sebastian Schaffert <se...@gmail.com> wrote:

> Hi all, 
> 
> some additions to Sergio's reply:
> 
> 
> 2014-07-07 8:11 GMT+02:00 Sergio Fernández <wi...@apache.org>:
> 
> 
> My main requirements are
> 
> 1. Sparql Query speed like native memory storage like Jena TDB.
> 
> We do not have performance figure about the KiWi triple store in comparison win Jena TDB or other. If you get some, please share it with the community to have a reference.
> 
> Please, take into account that KiWi is just the default triple store. In Marmotta you can easily use any Sesame-based triples store. Further details at:
> 
> http://marmotta.apache.org/platform/backends
> 
> Right now, SPARQL is NOT very fast in KiWi (compared to Virtuoso, but still faster than in some other Sesame backend). We are still working on improving SPARQL performance, but since SPARQL is by its very nature a very expressive query language not everything can be super-fast.
>  
> 4. Text Search
> 
> That feature was not coming to Marmotta from LMF (http://lmf.googlecode.com). But you should be able to still use it by adding this dependency to your webapp launcher:
> 
>   <dependency>
>     <groupId>at.newmedialab.lmf</groupId>
>     <artifactId>lmf-search</artifactId>
>     <version>3.2.0-SNAPSHOT</version>
>   </dependency>
> 
> All details at http://code.google.com/p/lmf/wiki/ModuleSemanticSearch
> 
> This is not necessarily needed. The KiWi SPARQL implementation since last version contains features for full-text search as part of SPARQL queries when using PostgreSQL:
> 
> http://marmotta.apache.org/kiwi/sparql.html
> 
> 
> Greetings,
> 
> Sebastian

Re: Marmotta as a linked data platform

Posted by Sebastian Schaffert <se...@gmail.com>.

Hi all,

some additions to Sergio's reply:


2014-07-07 8:11 GMT+02:00 Sergio Fernández <wi...@apache.org>:
>
>
>
>  My main requirements are
>>
>> 1. Sparql Query speed like native memory storage like Jena TDB.
>>
>
> We do not have performance figure about the KiWi triple store in
> comparison win Jena TDB or other. If you get some, please share it with the
> community to have a reference.
>
> Please, take into account that KiWi is just the default triple store. In
> Marmotta you can easily use any Sesame-based triples store. Further details
> at:
>
> http://marmotta.apache.org/platform/backends


Right now, SPARQL is NOT very fast in KiWi (compared to Virtuoso, but still
faster than in some other Sesame backend). We are still working on
improving SPARQL performance, but since SPARQL is by its very nature a very
expressive query language not everything can be super-fast.


>  4. Text Search
>>
>
> That feature was not coming to Marmotta from LMF (
> http://lmf.googlecode.com). But you should be able to still use it by
> adding this dependency to your webapp launcher:
>
>   <dependency>
>     <groupId>at.newmedialab.lmf</groupId>
>     <artifactId>lmf-search</artifactId>
>     <version>3.2.0-SNAPSHOT</version>
>   </dependency>
>
> All details at http://code.google.com/p/lmf/wiki/ModuleSemanticSearch


This is not necessarily needed. The KiWi SPARQL implementation since last
version contains features for full-text search as part of SPARQL queries
when using PostgreSQL:

http://marmotta.apache.org/kiwi/sparql.html


Greetings,

Sebastian

Re: Marmotta as a linked data platform

Posted by Sergio Fernández <se...@salzburgresearch.at>.

Hi Sebastian,

On 08/07/14 09:53, Sebastian Schaffert wrote:
> Since we are serving all resources through our own filter, I don't think it
> is a Tomcat issue. Thanks to Joshua for pointing out this problem, it
> should be easy to fix this way. Unfortunately, mime-util is not very much
> maintained, so maybe we should find a different way to identify the MIME
> type of a file (Tika?).

Right, it might come from our resource filter implementation...

> Is there already a Jira issue for this topic?

Yes, MARMOTTA-499. I've just pushed a new topic branch with a tentative 
solution based on both Java NIO and Apache Tika instead of mime-utils.

Since it is not so simple to reproduce, Joshua, do you think you can 
help us to debug it? You just need to:

$ git clone -b MARMOTTA-499 
https://git-wip-us.apache.org/repos/asf/marmotta.git MARMOTTA-499
$ cd MARMOTTA-499
$ mvn clean install -DskipTests
$ cd launchers/marmotta-webapp
$ mvn tomcat7:run

And then check at http://localhost:8080/ if in your system styles are 
correctly rendered with the new implementation of mimetype detection.

Thanks so much for your collaboration.

Cheers,

-- 
Sergio Fernández
Senior Researcher
Knowledge and Media Technologies
Salzburg Research Forschungsgesellschaft mbH
Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria
T: +43 662 2288 318 | M: +43 660 2747 925
sergio.fernandez@salzburgresearch.at
http://www.salzburgresearch.at

Re: Marmotta as a linked data platform

Posted by Sebastian Schaffert <se...@gmail.com>.

Hi Sergio,

Since we are serving all resources through our own filter, I don't think it
is a Tomcat issue. Thanks to Joshua for pointing out this problem, it
should be easy to fix this way. Unfortunately, mime-util is not very much
maintained, so maybe we should find a different way to identify the MIME
type of a file (Tika?). Is there already a Jira issue for this topic?

Greetings,

Sebastian


2014-07-08 7:08 GMT+02:00 Sergio Fernández <
sergio.fernandez@salzburgresearch.at>:

> I guess the CSS issue is related to MARMOTTA-499:
>
>   https://issues.apache.org/jira/browse/MARMOTTA-499
>
> Basically Tomcat sends all .css files with the wrong mime-type
> (application/x-pointplus), and then styles are not interpreted by the
> browser.
>
> Are you using Mac OS X too? Alvaro Graces (CCed) reported the same issue
> few months ago, and I have to say we do not have a solution so far. The
> weird thing is that we cannot reproduce it: it works fine in some OSX
> (tested with Rupert and Thomas), but fails in some others. In GNU/Linux I
> never had such issue.
>
> I'll report it to Apache Tomcat, maybe they have a solution...
>
> Cheers,
>
>
>
> On 07/07/14 19:38, Joshua Dunham wrote:
>
>> Hi Prashant,
>>
>>     I’m a new user as well so I’m following this closely. One thing I
>> didn’t see answered yet was your concern regarding the CSS not rendering. I
>> also had this (so it’s good I’m not the only one!). The problem is in the
>> mime-util-2.1.3.jar file in the WEB-INF folder under marmotta tomcat web
>> apps folder. Inside of this it’s defined that text/css has a mime-type of
>> x-pointplus. I’m not sure what that is but it can be fixed.
>>
>>     If you take this jar out, extract it (jar xf mime-util-2.1.3.jar),
>> find the mime-util.properties text file, edit the css entry (I removed the
>> point plus bit since i have never used, but I think you could reverse the
>> order) and update the file in the jar (jar uf mime-util-2.1.3.jar
>> /path/to/mime-util.properties) it works. Please check syntax since I’m
>> writing this from memory.
>>
>>   Best,
>> -Joshua
>>
>>
>> Joshua Dunham
>> Scientific Computing | Vertex
>> E: joshua_dunham@vrtx.com<ma...@vrtx.com>
>> P: 617-229-5157
>> W: http://www.vrtx.com<http://www.vrtx.com/>
>>
>> L: http://linkedin.com/in/joshuadunham
>> S: joshua.dunham
>>
>> On Jul 7, 2014, at 6:35 AM, Prashant <pnema@icloud.com<mailto:pnema
>> @icloud.com>> wrote:
>>
>> Thanks Sergio
>> Yes I was able to build from latest source. I will file JIRA for admin
>> interface later today or tomorrow. But in brief following were the issues.
>>
>> 1. No style sheet in admin ui. It just comes as plain text. May be some
>> issues with my installation! Very time consuming to navigate in plain text
>> ui.
>> 2. Sparql interface does not support construct queries ( may be this is
>> feature but not urgent)
>> 3. Login issue as you mention.
>> 4. Data import failed number of times ( I tried all sort by file upload,
>> dropping in import directory etc). Large file may be the problem
>> 5. Status under task status is not showing proper status.
>>
>>
>>
>> Thanks
>> Prashant
>>
>>
>>
>>
>>
>> On Jul 7, 2014, at 7:11 AM, Sergio Fernández <wikier@apache.org<mailto:
>> wikier@apache.org>> wrote:
>>
>> Hi,
>>
>> On 02/07/14 15:16, Prashant wrote:
>> I am planning to use Marmotta linked data platform in production to build
>> applications for my client.
>>
>> Great! Let's see where we can help you.
>>
>> My main requirements are
>>
>> 1. Sparql Query speed like native memory storage like Jena TDB.
>>
>> We do not have performance figure about the KiWi triple store in
>> comparison win Jena TDB or other. If you get some, please share it with the
>> community to have a reference.
>>
>> Please, take into account that KiWi is just the default triple store. In
>> Marmotta you can easily use any Sesame-based triples store. Further details
>> at:
>>
>> http://marmotta.apache.org/platform/backends<https://
>> urldefense.proofpoint.com/v1/url?u=http://marmotta.apache.
>> org/platform/backends&k=nlnsHLn0ybmZ0IJA5X3Bfg%3D%3D%0A&r=
>> Ti2ZOdevHxjcPRKaSErKsJsdngV4bllmizMzlKHBb0U%3D%0A&m=chl1fibmY3xgb2Lu%
>> 2Bivk7jTFqzs8ZD5HSLy95D7aZhE%3D%0A&s=07e3f1cc4e04b52984438778957c1a
>> 9a48d480d099b45635ed84783b84c0966f>
>>
>>
>> 2. Versioning so that user can get snapshot of data at specified date by
>> sparql queries.
>>
>> Such feature is available in KiWi:
>>
>> http://marmotta.apache.org/platform/versioning-module
>>
>> 4. Text Search
>>
>> That feature was not coming to Marmotta from LMF (
>> http://lmf.googlecode.com<http://lmf.googlecode.com/>). But you should
>> be able to still use it by adding this dependency to your webapp launcher:
>>
>>
>>   <dependency>
>>     <groupId>at.newmedialab.lmf</groupId>
>>     <artifactId>lmf-search</artifactId>
>>     <version>3.2.0-SNAPSHOT</version>
>>   </dependency>
>>
>> All details at http://code.google.com/p/lmf/wiki/ModuleSemanticSearch
>>
>> 5. Internationalization
>>
>> What do you mean with internationalization? I think that depends on the
>> concrete application you want to build on top of, and the use the
>> internationalization capabilities that RDF provides.
>>
>> I was playing with binary of Apache Marmotta 3.2.1 and found numerous
>> issues while doing administration. It appears that it has been published in
>> May 2013. Since then there is no major release. So I moved to latest git
>> source and I found it has lots of good integration with cache, different
>> backend like titan etc.
>>
>> Well, the date issue was already addressed by Nandana.
>>
>> We are aware of some login issues in the admin ui. But if you could
>> detail your problems as issues at Jira, that would be great.
>>
>> My questions here are
>>
>> 1. How stable is current source code is? can it be used in production
>> like setup.
>> 2. Is there any documentation to build and release current source code?
>>
>> I'd say the current source code (3.3.0-SNAPSHOT is pretty stable). Try to
>> build it by youselft as described at:
>>
>> http://marmotta.apache.org/installation#source
>>
>> and tell us about it.
>>
>> Thanks so much for the feedback/
>>
>> Cheers,
>>
>> --
>> Sergio Fernández
>> Partner Technology Manager
>> Redlink GmbH
>> m: +43 660 2747 925
>> e: sergio.fernandez@redlink.co<ma...@redlink.co>
>> w: http://redlink.co<http://redlink.co/>
>>
>>
>> This email message and any attachments are confidential and intended for
>> use by the addressee(s) only. If you are not the intended recipient, please
>> notify me immediately by replying to this message, and destroy all copies
>> of this message and any attachments. Thank you.
>>
>>
> --
> Sergio Fernández
> Senior Researcher
> Knowledge and Media Technologies
> Salzburg Research Forschungsgesellschaft mbH
> Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria
> T: +43 662 2288 318 | M: +43 660 2747 925
> sergio.fernandez@salzburgresearch.at
> http://www.salzburgresearch.at
>

Re: MARMOTTA-499

Posted by Sergio Fernández <wi...@apache.org>.

Joshua, Alvaro, not sure if you are subscribe to jira updates. So please 
check if the last news there bring some light to this issue:

https://issues.apache.org/jira/browse/MARMOTTA-499?focusedCommentId=14071564&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14071564

-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 660 2747 925
e: sergio.fernandez@redlink.co
w: http://redlink.co