You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@marmotta.apache.org by Fabian Cretton <Fa...@hevs.ch> on 2014/09/02 09:06:47 UTC
New module similar to LDCache
Hi,
I would like to implement a module that is similar to the LDCache
(following the previous discussions with Sergio about the overLOD
project).
I am currently reading about the LDCache functionalities here
http://marmotta.apache.org/ldcache/
and having a look at the code.
(It is a pretty steep curve for me to apprehend the Marmotta project,
but I think it is worse it instead of starting from scratch)
As I am new to this kind of project infrastructure, is there anything I
should read to better understand the all framework ? Maybe Java JEE
tutorials as the project description says "The Apache Marmotta Platform
is implemented as a light-weight Service-Oriented Architecture (SOA)
using the CDI/Weld service framework (i.e. the core components of Java
EE 6). "
Then, to create the new module, would it be a good idea to duplicate
the LDCache files (libraries and platform I guess) and modify them, or
should I better start from a new empty module as described here:
http://wiki.apache.org/marmotta/Customizing#Modules
Thank you for any help
Fabian
>>> Sergio Fernández<wi...@apache.org> 27.08.2014 16:31 >>>
Hi Fabian,
On 27/08/14 14:49, Fabian Cretton wrote:
> My first goal was: to build the all project locally, run my locally
> built Marmotta, and then start adding components.
> But my first concern now that I am digging deeper, is that Marmotta
is
> a pretty big project (about 80 projects), and so you might recommand
me
> not to import the main "pom.xml" in my eclipse environment, but
start
> smaller ?
Then start from the platform modules.
> If there is already a documentation about how to procede, thank you
to
> point me there, I didn't find any by myself.
Well, the overall build process is entirely manage by Maven, check
http://marmotta.apache.org/installation#source
> Nevertheless, I do have problems and errors in Eclipse, and hope you
> can help me about that.
Eclipse should be able to manage such king of size of modules with
Maven.
> The first problems I do have, are with many "Plugin execution not
> covered by lifecycle configuration" errors.
Some plugin lifecycles might not be supported inside Eclipse. Just
ignore it, you should not need them.
> Than I do have 6-7 : "Project build error: Non-resolvable parent
POM:
> Could not find artifact
> org.apache.marmotta:marmotta-parent:pom:3.2.1-SNAPSHOT and
> 'parent.relativePath' points at wrong local POM pom.xml
> /marmotta-backend-sparql line 23 Maven pom Loading Problem"
> and here I am pretty confused: it seems that some POM files are not
> up-to-date in this 3.3.0 current version, as they do still point to
a
> "3.2.1" parent POM file, but the parent is already in its "3.3.0"
> version ?
Sorry for the error. Those modules are out of the default profile, so
the release plugin did not update the versions accordingly. It's
already
fixed in the develop branch; please update your fork.
> Then, apart from those Maven errors, I do have a few java errors
with
> many "imports" or "types" which can't be resolved, and this seems
very
> strange to me. But maybe solving the main Maven problems here above
> would correct that ?
All dependencies are available from Maven central. Try to make a "maven
install" from the root.
> A first goal for me would be to update the Marmotta's main menu so
that
> under "Others", next to "Linked Data Caching", I could have a
"External
> Data Sources" menu and then work an that new module as discussed
earlier
> with you.
Then you need to create a custom module and add it to your custom
webapp
launcher. All the process is supported by Maven artifacts, as described
at:
http://wiki.apache.org/marmotta/Customizing#Modules
Hope that helps.
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 660 2747 925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Rép. : Marmotta 3.2.1 - Having problem to trigger the LDCache
Posted by Fabian Cretton <Fa...@hevs.ch>.
One more information.
Today I tried from scratch, reinstalling Marmotta, creating the end-point from the load sample.
And then querying "http://dbpedia.org/resource/Europe" with LDPath.
LDPath result was: resource http://dbpedia.org/resource/Europe does not exist
But I had one error message in the console:
07:38:52.962 ERROR - the HTTP request failed (status: HTTP/1.1 406 Unacceptable)
07:38:52.963 ERROR - HTTP client error while trying to retrieve resource http://dbpedia.org/resource/Europe: the HTTP request failed (status: HTTP/1.1 406 Unacceptable)
After that I tried again, but this message didn't come up again.
I don't think it comes from some firewall settings we have internally, but I'll check about that.
Thank you for any help
Fabian
>>> "Fabian Cretton" <Fa...@hevs.ch> 08.09.2014 15:50 >>>
Hi,
Today I tried without success to setup the LDCache from the Marmotta interface.
A first thing is that it is not clearly stated, in my opinion, when the LDCache is triggered.
The only further information I found was here:
http://markmail.org/message/m3ft65du5kbuzy5z#query:+page:1+mid:j347sjyegwshvhc7+state:results
and thus I understood that the SPARQL interface don't interact with LDCache, but the LDPath functionality does.
Is there anything else then running a LPath that would trigger the cache ?
First of all, I see this error when launching Marmotta 3.2.1, could it be related to my problems ?
INFO: Server startup in 11332 ms
15:25:45.495 WARN - pattern (C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).* is not a valid regular expression; disabling reader/writer filesystem (message was Illegal/unsupported escape sequence near index 4
(C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).* ^)
My "ldcache.enabled" parameter is checked.
I guess my problem is with setting up the end-point correctly.
I first tried the provided sample "load sample -> dbpedia", which gives in the interface:
kind: ld cache
Prefix: http://dbpedia\.org.*
Endpoint: http://dbpedia.org/sparql
But a few things are already not clear for me:
* kind: ld cache
When referring to a sparql end-point, shouldn't the kind be "sparkl" ? (I did try changing this, but with no success)
By the way, what is the meaning of the "cache" mode, I can't understand from reading the explanation. Is it a server that servers "linked data" ? if so, what is the difference with "linked data".
* prefix:
What are the "rules" to define those prefixes ? for instance I see in the example that "." are escaped ?
Defining a prefix as "http://dbpedia\.org ( http://dbpedia/.org ).*" will thus mean that a URI as http://dbpedia.org/resource/Europe is bound to that end-point, right ?
*endpoint:
The explanation says that for a SPARQL end-point, some parameters have to be specified:
"http://sparql.sindice.com/sparql?default-graph-uri=&query={query}&format={contenttype}"
Is this mandatory ? (as the provided sample don't specify them but just specifies "http://dbpedia.org/sparql"), are there default values ?
*mimetype:
Not filled by the sample, are their default values depending on the "mode of operation" ?
If it is possible to get a few full examples of those settings, for the different operation modes, it would be very helpful I guess.
Than, after trying different configurations, I tried from the LDPath interface to add the URI "http://dbpedia.org/resource/Europe" and the LDPath "rdfs:label :: xsd:string".
When hitting "Test", a message appears next to the button: resource http://dbpedia.org/resource/Europe does not exist.
It seems I don't see any corresponding message neither in the console nor in the marmotta-main.log.
I tried so many different configurations, on different SPARQL end-point, also trying a "linked data" as DBPedia does provide the content negociation mecanism, but without success.
Finally I tried to run a:
curl http://wh55650:8080/marmotta/cache/live?uri=http://dbpedia.org/resource/Europe
and it did return the rdf/xml file. But what I discovered then is that even without configuring any 'end-point' in Marmotta, this command was succeeding every time, so I think this command is not a sign that my end-point is well configured ? Except maybe if I was dealing with data that are not provided as derefencable "linked data" ?
I also tried on my local version 3.3.0, but with the same results. And also with the same WARNING when starting Marmotta.
Thank you for any help.
Fabian
Re: Marmotta 3.2.1 - Having problem to trigger the LDCache
Posted by Jakob Frank <ja...@apache.org>.
Hi Fabian,
On 2014-09-08 15:50, Fabian Cretton wrote:
> A first thing is that it is not clearly stated, in my opinion, when the
> LDCache is triggered.
>
> The only further information I found was here:
> http://markmail.org/message/m3ft65du5kbuzy5z#query:+page:1+mid:j347sjyegwshvhc7+state:results
> and thus I understood that the SPARQL interface don't interact with
> LDCache, but the LDPath functionality does.
> Is there anything else then running a LPath that would trigger the cache ?
LDCache is triggered everytime a triple-pattern like "<uri> * *" is
requested from the triple store. Then LDCache tries to load <uri> from
remote.
LDPath is using such queries quite often, SPARQL only in few cases
because many requests are directly issued to the database for performance.
Even viewing the resource in the resource browser
(http://wh55650:8080/marmotta/resource/*) should trigger LDCaching.
> First of all, I see this error when launching Marmotta 3.2.1, could it
> be related to my problems ?
> INFO: Server startup in 11332 ms
> 15:25:45.495 WARN - pattern
> (C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).*
> is not a valid regular expression; disabling reader/writer filesystem
> (message was Illegal/unsupported escape sequence near index 4
> (C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).*
> ^)
As the log says, it's only a warning and should not affect you unless
you are storing (binary) content in Marmotta.
Background and workaround for this issue was discussed on
users@marmotta.a.o [1]
> My "ldcache.enabled" parameter is checked.
>
> I guess my problem is with setting up the end-point correctly.
>
> I first tried the provided sample "load sample -> dbpedia", which gives
> in the interface:
> kind: ld cache
> Prefix: http://dbpedia\.org.*
> Endpoint: http://dbpedia.org/sparql
Looks like there is an error in the sample: Provider (kind) "ld cache"
does not work with an SPARQL endpoint - for that you need to select
"sparql" as kind.
But as you noticed: since DBPedia provides valid Linked Data, no extra
configuation is required - just request the resource and it will be fetched:
<http://wh55650:8080/marmotta/resource/?uri=http://dbpedia.org/resource/Europe>
> But a few things are already not clear for me:
> * kind: ld cache
> When referring to a sparql end-point, shouldn't the kind be "sparkl" ?
> (I did try changing this, but with no success)
> By the way, what is the meaning of the "cache" mode, I can't understand
> from reading the explanation. Is it a server that servers "linked data"
> ? if so, what is the difference with "linked data".
> * prefix:
> What are the "rules" to define those prefixes ? for instance I see in
> the example that "." are escaped ?
> Defining a prefix as "http://dbpedia\.org <http://dbpedia/.org>.*" will
> thus mean that a URI as http://dbpedia.org/resource/Europe is bound to
> that end-point, right ?
> *endpoint:
> The explanation says that for a SPARQL end-point, some parameters have
> to be specified:
> "http://sparql.sindice.com/sparql?default-graph-uri=&query={query}&format={contenttype}"
> Is this mandatory ? (as the provided sample don't specify them but just
> specifies "http://dbpedia.org/sparql"), are there default values ?
> *mimetype:
> Not filled by the sample, are their default values depending on the
> "mode of operation" ?
>
> If it is possible to get a few full examples of those settings, for the
> different operation modes, it would be very helpful I guess.
Sergio just now updated the documentation for the LDCache module [2],
please have a look and check if anything is missing or unclear.
Best,
Jakob
[1]
http://mail-archives.apache.org/mod_mbox/marmotta-users/201409.mbox/%3C730127A8D040CB4A976BE07F089997068BCBB097@EXC-MBX03.tsn.tno.nl%3E
[2] http://marmotta.apache.org/platform/ldcache-module.html
Marmotta 3.2.1 - Having problem to trigger the LDCache
Posted by Fabian Cretton <Fa...@hevs.ch>.
Hi,
Today I tried without success to setup the LDCache from the Marmotta interface.
A first thing is that it is not clearly stated, in my opinion, when the LDCache is triggered.
The only further information I found was here:
http://markmail.org/message/m3ft65du5kbuzy5z#query:+page:1+mid:j347sjyegwshvhc7+state:results
and thus I understood that the SPARQL interface don't interact with LDCache, but the LDPath functionality does.
Is there anything else then running a LPath that would trigger the cache ?
First of all, I see this error when launching Marmotta 3.2.1, could it be related to my problems ?
INFO: Server startup in 11332 ms
15:25:45.495 WARN - pattern (C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).* is not a valid regular expression; disabling reader/writer filesystem (message was Illegal/unsupported escape sequence near index 4
(C:\marmotta_inst\marmotta-home/resources|http://wh55650:8080/marmotta/resource/|urn:).* ^)
My "ldcache.enabled" parameter is checked.
I guess my problem is with setting up the end-point correctly.
I first tried the provided sample "load sample -> dbpedia", which gives in the interface:
kind: ld cache
Prefix: http://dbpedia\.org.*
Endpoint: http://dbpedia.org/sparql
But a few things are already not clear for me:
* kind: ld cache
When referring to a sparql end-point, shouldn't the kind be "sparkl" ? (I did try changing this, but with no success)
By the way, what is the meaning of the "cache" mode, I can't understand from reading the explanation. Is it a server that servers "linked data" ? if so, what is the difference with "linked data".
* prefix:
What are the "rules" to define those prefixes ? for instance I see in the example that "." are escaped ?
Defining a prefix as "http://dbpedia\.org ( http://dbpedia/.org ).*" will thus mean that a URI as http://dbpedia.org/resource/Europe is bound to that end-point, right ?
*endpoint:
The explanation says that for a SPARQL end-point, some parameters have to be specified:
"http://sparql.sindice.com/sparql?default-graph-uri=&query={query}&format={contenttype}"
Is this mandatory ? (as the provided sample don't specify them but just specifies "http://dbpedia.org/sparql"), are there default values ?
*mimetype:
Not filled by the sample, are their default values depending on the "mode of operation" ?
If it is possible to get a few full examples of those settings, for the different operation modes, it would be very helpful I guess.
Than, after trying different configurations, I tried from the LDPath interface to add the URI "http://dbpedia.org/resource/Europe" and the LDPath "rdfs:label :: xsd:string".
When hitting "Test", a message appears next to the button: resource http://dbpedia.org/resource/Europe does not exist.
It seems I don't see any corresponding message neither in the console nor in the marmotta-main.log.
I tried so many different configurations, on different SPARQL end-point, also trying a "linked data" as DBPedia does provide the content negociation mecanism, but without success.
Finally I tried to run a:
curl http://wh55650:8080/marmotta/cache/live?uri=http://dbpedia.org/resource/Europe
and it did return the rdf/xml file. But what I discovered then is that even without configuring any 'end-point' in Marmotta, this command was succeeding every time, so I think this command is not a sign that my end-point is well configured ? Except maybe if I was dealing with data that are not provided as derefencable "linked data" ?
I also tried on my local version 3.3.0, but with the same results. And also with the same WARNING when starting Marmotta.
Thank you for any help.
Fabian
Rép. : Re: New module similar to LDCache
Posted by Fabian Cretton <Fa...@hevs.ch>.
Thank you Sergio,
I know you are busy, but I hope we can clarify this these days, so that
we can come up with a clear idea of what we will developp, and how.
Here is what I understood and what I want to talk about: "All that
infrastructure is provided by the current LDCache module", and, if I get
it right, we would just have to implement new LDClients for new sources,
So there is something I don't understand here.
I'll try to clarify in more details the differences between
LDCLient\LDCache, and the functionality we want in overLOD, according to
my shallow understanding so far. I haven't played fully with
LDClient\LDCache so far, that is what I will do first thing next
monday.
* LDclient(s) are different clients for different data structures (aka
RDFizers), for instance one LDClient for RDFa, one for XML, etc. and
then also for specific data sources and their structures: one for
youtube data, one for facebook. The LDClient knows how to access the
data and how to import them in Marmotta, requiring some transformation
if needed (for instance translate XML to RDF, etc.). This is where we
would, in our project, implement a LDClient for microdata based on
schema.org.
But the LDClient is not specific to one source (maybe except for
facebook/youtube clients and the like).
In our case, we want to define a LDClient for a specific 'Kind' of
source, let say some rdf files, but then we want to define pointers to
very specific RDF files, for instance 20 of them. To me that would be
done in the higher level, maybe LDCache.
Or do you mean that we create a generic LDClient that is able to import
one kind of data, and then we instanciate 20 times LDClient, one for
each specific source ?
I give here examples for situations we want to handle (and certainly
mainy coming apps based on linked data):
- an way is defined to publish a catalog on the web (ontologies)
-> here, if 30 providers publish according to that way, we want to have
a reference to the 20 of them, and retrieve their data locally (cache)
- or: we could want to import different sets of data from an end-point:
the french cities from DBPedia, the elevations from geoNames, but not
only data where the resources are subject, they might be object too.
* Then the LDCache is an automatic and transparent functionality which
will, during queries on the triple store, see that information about a
resource can't be found in the triple store, but should be found on the
web in order to answer the query. That's what I understand from my
local page here: http://localhost:8080/marmotta/cache/admin/about.html.
and from the LDCache descriptions I found here:
http://marmotta.apache.org/ldcache/index.html.
In order for the LDCache to work, the administrator has to define
LD-Cache endpoints. The LD-Cache will rely on the ressources "prefix" in
order to know which resource to find on which endpoint.
Another information that is different from what we want: "SPARQL
(access to a resource via a SPARQL endpoint): retrieves the triples of a
resource from a defined SPARQL endpoint by issuing a query for all
triples with the resource as subject"
-> we are not solely interested in triples where the resource is
subject
Finally, the LD-Cache will save the triples concerning the resource in
the LD-Cache context. I guess all the triples retrieved by
LD-Cache\LD-Client are saved in a same and single context ?
If so, I do have a question here: how does LD cache identify which
triples in one single cache context needs to be updated ? -> maybe it
takes all the triples which have a specific resource as subject ? or the
timeout is specific to an endpoint ?
And so, the functionalities I did describe in my former email, which I
will further developp here, seems to me not implemented yet, and of
great importance for further use of linked data outside of "research"
projects:
Firstly, define precisely data we want to cache (RDF or any data
handled by a LDClient) to be cached in the server.
It will not be the user SPARQL queries that influence the cached data
(which seems the case with LDCache), but an administrator can very
specifically choose which data to cache (and have control them) -> then
only those validated data are available for the end-user apps based on
that instance of Marmotta.
Then we want to have for instance the all content of a file, or the
result of a SPARQL Construct
-> and not just triples where a resource is subject
and, to manage the update of those triples, I think we need one context
per source (otherwise how to know which triples where removed from the
source, etc..)
This seems to me very pretty different from the current management of
LD-Cache endpoints.
Another exemple: the LD-Cache endpoints allows to say that resources
with the prefix http://dbpedia.org/resource/
should be found on a specific server. When such a resource is met (in a
SPARQL query or maybe in triples uploaded in the store ?), the LDCache
will query the DBPedia end-point and retrieve all triples where this
resource is subject.
In overLOD: we might want to configure the referencer, so that a SPARQL
construct is run on DBPedia to retrieve all the cities from a specific
country, only the city and its label in french and english (not all
labels!), its population, but also an information where the city is
object of a triple (which might be needed when no inverse properties are
defined)
->those triples will be savec in a specific context
-> they need to be validated
-> we need to know when those information have changed on the server,
and update the cache.
So we can't just set a 'timeout', as we can cache some files that are
never updated (and so no need to reload the data), but also some other
files which are regularly updated.
This 'update' mechanism is a functionality I would hope to talk about
with the Marmotta team, it seems to me more efficient then the current
LD-Cache and its timeout, but I am not sure yet.
Aren't there some big difference here, eventhoug the background is
similar ?
Are those functionalities part of the LDClient (as you suggested) ?
Seems to me some could be implemented in LDClient, but some others in
LDCache (or a new LDCache created for overLOD, which I called last time
'External Data Sources').
Thank you to help me move forward
Fabian
>>> Sergio Fernández<wi...@apache.org> 05.09.2014 09:36 >>>
Hi Fabian,
On 02/09/14 14:09, Fabian Cretton wrote:
> So that would be the goal of the "External data sources" module,
which
> was originaly called "overLOD Referencer" in the document [1]:
> - define precisely RDF data to be cached in the server: that could be
a
> RDF File, a SPARQL CONSTRUCT on a end-point, etc.
> - find a way to validate the content of that data -> here we might
not
> want to reason in an open world assumption, but if a property is
defined
> with a certain range, we would want to check that the objects in the
> file ARE effectively instances from that defined class (for instance
> using SPARQL queries to validate the content, instead of a
reasoner).
> - find a way to manage automatically the updates: it could be a
'pull'
> from Marmotta depending on some VoID data provided by the source, or
the
> source could put in place a "ping" to marmotta, RSS-like features,
like
> it was done by Ping-The-Semantic-Web or Sindice
All that infrastructure is provided by the current LDCache module. If I
got it right, where you actually need to plug-in into this
infrastructure is at the LDClient level:
* you can define new LDClient Data Providers for your specific
sources
* which can wrap all the validation logic you need
* then LDCache will transparently make use of your LDClient
provider
* to avoid conflicts with the default providers, they can be
disabled
If that setup fit with your ideas and needs, I'd recommend you to take
a
look to the current providers:
https://github.com/apache/marmotta/tree/master/libraries/ldclient
Some of them just do data lifting from others formats (e.g., XML), some
wrap APIs to get RDF out of them (e.g., Facebook), and some do other
kind of validations and fixes (e.g., the Freebase provides does RDF
syntax fixing before parsing).
Hope that helps. I guess we have to provide better documentations and
diagrams to understand the infrastructure LDClient+LDCache provide.
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 660 2747 925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: New module similar to LDCache
Posted by Sergio Fernández <wi...@apache.org>.
Hi Fabian,
On 02/09/14 14:09, Fabian Cretton wrote:
> So that would be the goal of the "External data sources" module, which
> was originaly called "overLOD Referencer" in the document [1]:
> - define precisely RDF data to be cached in the server: that could be a
> RDF File, a SPARQL CONSTRUCT on a end-point, etc.
> - find a way to validate the content of that data -> here we might not
> want to reason in an open world assumption, but if a property is defined
> with a certain range, we would want to check that the objects in the
> file ARE effectively instances from that defined class (for instance
> using SPARQL queries to validate the content, instead of a reasoner).
> - find a way to manage automatically the updates: it could be a 'pull'
> from Marmotta depending on some VoID data provided by the source, or the
> source could put in place a "ping" to marmotta, RSS-like features, like
> it was done by Ping-The-Semantic-Web or Sindice
All that infrastructure is provided by the current LDCache module. If I
got it right, where you actually need to plug-in into this
infrastructure is at the LDClient level:
* you can define new LDClient Data Providers for your specific sources
* which can wrap all the validation logic you need
* then LDCache will transparently make use of your LDClient provider
* to avoid conflicts with the default providers, they can be disabled
If that setup fit with your ideas and needs, I'd recommend you to take a
look to the current providers:
https://github.com/apache/marmotta/tree/master/libraries/ldclient
Some of them just do data lifting from others formats (e.g., XML), some
wrap APIs to get RDF out of them (e.g., Facebook), and some do other
kind of validations and fixes (e.g., the Freebase provides does RDF
syntax fixing before parsing).
Hope that helps. I guess we have to provide better documentations and
diagrams to understand the infrastructure LDClient+LDCache provide.
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 660 2747 925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Rép. : Re: New module similar to LDCache
Posted by Fabian Cretton <Fa...@hevs.ch>.
Jakob,
Thank you for your quick answer.
This discussion is important for me, I hope we can clarify things
together. If you confirm that this feature is a plus for Marmotta, then
I could work on it, and we could take decisions together about the best
ways to implement the different functionalities.
The goal of the functionality, as described in the document I did
prepare to talk with the Marmotta team [1], seems to me different then
the LDCache, eventhough pretty similar.
That goal would be to setup a triple store for a specific purpose, and
build apps over those "controlled and validated" data. But the data
would mainly come from external sources, distributed sources. For
instance, when creating an app for engineers in the building field, we
could want to base that app on data coming from different building
material providers. Those providers publishing their catalog in RDF
(publishing is not our concern in this project). They can publish it as
an .rdf file, RDFa directly on their webside, or even a sparql
end-point. We then define the data sources we want to include (those
catalogues for instance), and the system helps an administrator to
validate those data (they should not contain unwanted or unexpected
data) and keep them up-to-date (as soon as the original data is updated,
the system must know it and do something automatically or
semi-automatically).
As I understand the LDCache, it is a functionality to transparently
cache data from the LOD when a triple contains a reference to a URI that
can be reached by one of the defined "LD Cache Endpoints". There is not
much control about which information is precisely retrived, how to
validate the content, or how that information is automatically updated
(I don't know yet how the expiry time is handeld).
This functionality, in our opinion, is a mandatory functionality to
bring the LOD to its full potential, for real world applications (and
not just for research purpose) -> here you need to know which data you
work on, know they are reliable, etc.
So that would be the goal of the "External data sources" module, which
was originaly called "overLOD Referencer" in the document [1]:
- define precisely RDF data to be cached in the server: that could be a
RDF File, a SPARQL CONSTRUCT on a end-point, etc.
- find a way to validate the content of that data -> here we might not
want to reason in an open world assumption, but if a property is defined
with a certain range, we would want to check that the objects in the
file ARE effectively instances from that defined class (for instance
using SPARQL queries to validate the content, instead of a reasoner).
- find a way to manage automatically the updates: it could be a 'pull'
from Marmotta depending on some VoID data provided by the source, or the
source could put in place a "ping" to marmotta, RSS-like features, like
it was done by Ping-The-Semantic-Web or Sindice
Please refer to [1] for more detailed information, and let me know if
the purpose of this is really not clear ?
I hope you will be able to tell me if I did misunderstand LDCache and
finally it can play that exact role ?
If LDCache can not do that right now, do you think I should work on a
new module, or just add some functionalities to LDCache ?
Hope we can have an interesting discussion
Thank you for your help
Fabian
[1]
https://dl.dropboxusercontent.com/u/852552/Marmotta_OverLOD%20Surfer%20presentation_0.2.pdf
>>> Jakob Frank <ja...@apache.org> 02.09.2014 12:45 >>>
Hi Fabian,
looks like you chose a big one for starting ;-)
LDCache plugs into the Sesame-Sail stack to automatically retrieve
remote resources that are available in the local triple store.
Sesame does not use CDI but the build-in Java ServiceLoader [1], so
plugging in there is not as easy.
On the other hand: why do you want to implement module similar to
LDCache? What feature do you need that can't be solved using LDCache -
for me, your "exteral datasource" module sounds exactly like LDCache
in action...
Best,
Jakob
[1]
http://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html
On 2 September 2014 09:06, Fabian Cretton <Fa...@hevs.ch>
wrote:
> Hi,
>
> I would like to implement a module that is similar to the LDCache
(following
> the previous discussions with Sergio about the overLOD project).
> I am currently reading about the LDCache functionalities here
> http://marmotta.apache.org/ldcache/
> and having a look at the code.
>
> (It is a pretty steep curve for me to apprehend the Marmotta project,
but I
> think it is worse it instead of starting from scratch)
>
> As I am new to this kind of project infrastructure, is there anything
I
> should read to better understand the all framework ? Maybe Java JEE
> tutorials as the project description says "The Apache Marmotta
Platform is
> implemented as a light-weight Service-Oriented Architecture (SOA)
using the
> CDI/Weld service framework (i.e. the core components of Java EE 6).
"
>
> Then, to create the new module, would it be a good idea to duplicate
the
> LDCache files (libraries and platform I guess) and modify them, or
should I
> better start from a new empty module as described here:
> http://wiki.apache.org/marmotta/Customizing#Modules
>
> Thank you for any help
> Fabian
>
>
>>>> Sergio Fernández<wi...@apache.org> 27.08.2014 16:31 >>>
> Hi Fabian,
>
> On 27/08/14 14:49, Fabian Cretton wrote:
>> My first goal was: to build the all project locally, run my locally
>> built Marmotta, and then start adding components.
>> But my first concern now that I am digging deeper, is that Marmotta
is
>> a pretty big project (about 80 projects), and so you might recommand
me
>> not to import the main "pom.xml" in my eclipse environment, but
start
>> smaller ?
>
> Then start from the platform modules.
>
>> If there is already a documentation about how to procede, thank you
to
>> point me there, I didn't find any by myself.
>
> Well, the overall build process is entirely manage by Maven, check
> http://marmotta.apache.org/installation#source
>
>> Nevertheless, I do have problems and errors in Eclipse, and hope
you
>> can help me about that.
>
> Eclipse should be able to manage such king of size of modules with
Maven.
>
>> The first problems I do have, are with many "Plugin execution not
>> covered by lifecycle configuration" errors.
>
> Some plugin lifecycles might not be supported inside Eclipse. Just
> ignore it, you should not need them.
>
>> Than I do have 6-7 : "Project build error: Non-resolvable parent
POM:
>> Could not find artifact
>> org.apache.marmotta:marmotta-parent:pom:3.2.1-SNAPSHOT and
>> 'parent.relativePath' points at wrong local POM pom.xml
>> /marmotta-backend-sparql line 23 Maven pom Loading Problem"
>> and here I am pretty confused: it seems that some POM files are not
>> up-to-date in this 3.3.0 current version, as they do still point to
a
>> "3.2.1" parent POM file, but the parent is already in its "3.3.0"
>> version ?
>
> Sorry for the error. Those modules are out of the default profile,
so
> the release plugin did not update the versions accordingly. It's
already
> fixed in the develop branch; please update your fork.
>
>> Then, apart from those Maven errors, I do have a few java errors
with
>> many "imports" or "types" which can't be resolved, and this seems
very
>> strange to me. But maybe solving the main Maven problems here above
>> would correct that ?
>
> All dependencies are available from Maven central. Try to make a
"maven
> install" from the root.
>
>> A first goal for me would be to update the Marmotta's main menu so
that
>> under "Others", next to "Linked Data Caching", I could have a
"External
>> Data Sources" menu and then work an that new module as discussed
earlier
>> with you.
>
> Then you need to create a custom module and add it to your custom
webapp
> launcher. All the process is supported by Maven artifacts, as
described at:
>
> http://wiki.apache.org/marmotta/Customizing#Modules
>
> Hope that helps.
>
> Cheers,
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 660 2747 925
> e: sergio.fernandez@redlink.co
> w: http://redlink.co
Re: New module similar to LDCache
Posted by Jakob Frank <ja...@apache.org>.
Hi Fabian,
looks like you chose a big one for starting ;-)
LDCache plugs into the Sesame-Sail stack to automatically retrieve
remote resources that are available in the local triple store.
Sesame does not use CDI but the build-in Java ServiceLoader [1], so
plugging in there is not as easy.
On the other hand: why do you want to implement module similar to
LDCache? What feature do you need that can't be solved using LDCache -
for me, your "exteral datasource" module sounds exactly like LDCache
in action...
Best,
Jakob
[1] http://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html
On 2 September 2014 09:06, Fabian Cretton <Fa...@hevs.ch> wrote:
> Hi,
>
> I would like to implement a module that is similar to the LDCache (following
> the previous discussions with Sergio about the overLOD project).
> I am currently reading about the LDCache functionalities here
> http://marmotta.apache.org/ldcache/
> and having a look at the code.
>
> (It is a pretty steep curve for me to apprehend the Marmotta project, but I
> think it is worse it instead of starting from scratch)
>
> As I am new to this kind of project infrastructure, is there anything I
> should read to better understand the all framework ? Maybe Java JEE
> tutorials as the project description says "The Apache Marmotta Platform is
> implemented as a light-weight Service-Oriented Architecture (SOA) using the
> CDI/Weld service framework (i.e. the core components of Java EE 6). "
>
> Then, to create the new module, would it be a good idea to duplicate the
> LDCache files (libraries and platform I guess) and modify them, or should I
> better start from a new empty module as described here:
> http://wiki.apache.org/marmotta/Customizing#Modules
>
> Thank you for any help
> Fabian
>
>
>>>> Sergio Fernández<wi...@apache.org> 27.08.2014 16:31 >>>
> Hi Fabian,
>
> On 27/08/14 14:49, Fabian Cretton wrote:
>> My first goal was: to build the all project locally, run my locally
>> built Marmotta, and then start adding components.
>> But my first concern now that I am digging deeper, is that Marmotta is
>> a pretty big project (about 80 projects), and so you might recommand me
>> not to import the main "pom.xml" in my eclipse environment, but start
>> smaller ?
>
> Then start from the platform modules.
>
>> If there is already a documentation about how to procede, thank you to
>> point me there, I didn't find any by myself.
>
> Well, the overall build process is entirely manage by Maven, check
> http://marmotta.apache.org/installation#source
>
>> Nevertheless, I do have problems and errors in Eclipse, and hope you
>> can help me about that.
>
> Eclipse should be able to manage such king of size of modules with Maven.
>
>> The first problems I do have, are with many "Plugin execution not
>> covered by lifecycle configuration" errors.
>
> Some plugin lifecycles might not be supported inside Eclipse. Just
> ignore it, you should not need them.
>
>> Than I do have 6-7 : "Project build error: Non-resolvable parent POM:
>> Could not find artifact
>> org.apache.marmotta:marmotta-parent:pom:3.2.1-SNAPSHOT and
>> 'parent.relativePath' points at wrong local POM pom.xml
>> /marmotta-backend-sparql line 23 Maven pom Loading Problem"
>> and here I am pretty confused: it seems that some POM files are not
>> up-to-date in this 3.3.0 current version, as they do still point to a
>> "3.2.1" parent POM file, but the parent is already in its "3.3.0"
>> version ?
>
> Sorry for the error. Those modules are out of the default profile, so
> the release plugin did not update the versions accordingly. It's already
> fixed in the develop branch; please update your fork.
>
>> Then, apart from those Maven errors, I do have a few java errors with
>> many "imports" or "types" which can't be resolved, and this seems very
>> strange to me. But maybe solving the main Maven problems here above
>> would correct that ?
>
> All dependencies are available from Maven central. Try to make a "maven
> install" from the root.
>
>> A first goal for me would be to update the Marmotta's main menu so that
>> under "Others", next to "Linked Data Caching", I could have a "External
>> Data Sources" menu and then work an that new module as discussed earlier
>> with you.
>
> Then you need to create a custom module and add it to your custom webapp
> launcher. All the process is supported by Maven artifacts, as described at:
>
> http://wiki.apache.org/marmotta/Customizing#Modules
>
> Hope that helps.
>
> Cheers,
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 660 2747 925
> e: sergio.fernandez@redlink.co
> w: http://redlink.co