You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Olivier Grisel <ol...@ensta.org> on 2011/04/09 22:20:18 UTC
About the test suite in the kres profile
Hi all,
I am running the tests with "mvn clean install -Pkres" and I have too
issues (most likely related):
- they need internet access: this means they will randomly fail if the
remote resources are not available (network is down, remote server is
down, ...)
- some of them are much too slow: to ensuite people will regularly run
the tests before checking in changes into the
We should make each test take less than a couple of seconds to run and
avoid the network connection requirement.
The slow / internet tests are in ontonet and the rule manager.
For instance:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 248.584 sec
Running org.apache.stanbol.ontologymanager.ontonet.ontology.TestIndexing
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 245.178 sec
Running org.apache.stanbol.ontologymanager.ontonet.ontology.TestOntologySpaces
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.027 sec
Running org.apache.stanbol.rules.manager.LoadRuleFileTest
http://kres.iks-project.eu/ontology/meta/rmi.owl#
106 121 106 121
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.346 sec
Running org.apache.stanbol.rules.manager.RemoveRecipeTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.226 sec
Running org.apache.stanbol.rules.manager.RemoveRuleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.052 sec
Running org.apache.stanbol.rules.manager.GetRuleTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.485 sec
Running org.apache.stanbol.rules.manager.AddRecipeTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.664 sec
Running org.apache.stanbol.rules.manager.AddRuleTest
entriamo http://kres.iks-project.eu/ontology/meta/rmi.owl#MyRuleA
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.977 sec
Running org.apache.stanbol.rules.manager.GetRecipeTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.492 sec
Running org.apache.stanbol.rules.manager.RuleStoreTest
Possible solutions:
- make the rule engine have an offline mode where all the necessary
resources are preloaded from local resources included in the
src/test/resources folder of the project.
- change the ONManager tests to feature an offline mode where all
loaded ontology definitions are coming from local resources (e.g. in
src/test/resources).
- alternatively, use a mock network resource using one of the numerous
mock system to simulate network resources without downloading them.
This solution is less than optimal though: the rule manager has to be
fast in production, and should not relying on slow downloads anyway:
the default settings should be to use only locally pre-cached /
pre-loaded resources and only download missing resources on explicit
user requests rather than trying to do it automagically in the
background and randomly fail if the network is too slow.
- in the short term: exclude those tests from the automated surefire
test suite [1]
[1] http://maven.apache.org/plugins/maven-surefire-plugin/examples/inclusion-exclusion.html
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Rupert Westenthaler <ru...@gmail.com>.
Short update:
I created three issues covering this two points:
> (1) Referenced Sites will provide metadata so that callers can ask if
> they can operate offline. This can be used by the
> ReferencedSiteEntityEnhancementEngine to determine if it runs in
> Offline of Online mode.
> (2) I will create a small local index that only includes the entities
> needed for the unit tests. This can than be installed in the
> initialization of the integration tests. This feature is an
> requirement for creating Integration tests of the Entityhub. It will
> also allow to use the ReferencedSiteEntityEnhancementEngine with the
> integration test of the Enhancer.
- STANBOL-161 for adding Metadata to ReferencedSite
- STANBOL-162 to use this Information for the
ReferencedSiteEntityEnhancementEngine
- STANBOL-163 for ensuring that tests can use a customized data set
that does not need an external service.
best
Rupert Westenthaler
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
On Sun, Apr 10, 2011 at 9:27 AM, Fabian Christ
<ch...@googlemail.com> wrote:
> +1
>
> 2011/4/10 Olivier Grisel <ol...@ensta.org>:
>> 2011/4/10 Rupert Westenthaler <ru...@gmail.com>:
>>> Hi
>>>
>>> On Sat, Apr 9, 2011 at 10:25 PM, Olivier Grisel
>>> <ol...@ensta.org> wrote:
>>>> Actually this problem is not only occurring in the kres profile: I
>>>> just had a random failure in the BenchmarkTest that uses the full
>>>> launcher because the dbpedia referenced site got a random error 500
>>>> from dbpedia.org... even though the BenchmarkTest is using the offline
>>>> mode explicitly: it seems that the yard is somehow ignoring the
>>>> offline mode.
>>> Yes this is an issue I am aware of and it will be addressed by two things:
>>> (1) Referenced Sites will provide metadata so that callers can ask if
>>> they can operate offline. This can be used by the
>>> ReferencedSiteEntityEnhancementEngine to determine if it runs in
>>> Offline of Online mode.
>>> (2) I will create a small local index that only includes the entities
>>> needed for the unit tests. This can than be installed in the
>>> initialization of the integration tests. This feature is an
>>> requirement for creating Integration tests of the Entityhub. It will
>>> also allow to use the ReferencedSiteEntityEnhancementEngine with the
>>> integration test of the Enhancer.
>>>
>>> I hope I can provide both things within the coming two weeks. In the
>>> meanwhile we can deactivate the ReferencedSiteEntityEnhancementEngine
>>> by adding a Reference to Offline.
>>
>> Ok +1
>>
>> --
>> Olivier
>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>
>
>
>
> --
> Fabian
>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: About the test suite in the kres profile
Posted by Fabian Christ <ch...@googlemail.com>.
+1
2011/4/10 Olivier Grisel <ol...@ensta.org>:
> 2011/4/10 Rupert Westenthaler <ru...@gmail.com>:
>> Hi
>>
>> On Sat, Apr 9, 2011 at 10:25 PM, Olivier Grisel
>> <ol...@ensta.org> wrote:
>>> Actually this problem is not only occurring in the kres profile: I
>>> just had a random failure in the BenchmarkTest that uses the full
>>> launcher because the dbpedia referenced site got a random error 500
>>> from dbpedia.org... even though the BenchmarkTest is using the offline
>>> mode explicitly: it seems that the yard is somehow ignoring the
>>> offline mode.
>> Yes this is an issue I am aware of and it will be addressed by two things:
>> (1) Referenced Sites will provide metadata so that callers can ask if
>> they can operate offline. This can be used by the
>> ReferencedSiteEntityEnhancementEngine to determine if it runs in
>> Offline of Online mode.
>> (2) I will create a small local index that only includes the entities
>> needed for the unit tests. This can than be installed in the
>> initialization of the integration tests. This feature is an
>> requirement for creating Integration tests of the Entityhub. It will
>> also allow to use the ReferencedSiteEntityEnhancementEngine with the
>> integration test of the Enhancer.
>>
>> I hope I can provide both things within the coming two weeks. In the
>> meanwhile we can deactivate the ReferencedSiteEntityEnhancementEngine
>> by adding a Reference to Offline.
>
> Ok +1
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
--
Fabian
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/10 Rupert Westenthaler <ru...@gmail.com>:
> Hi
>
> On Sat, Apr 9, 2011 at 10:25 PM, Olivier Grisel
> <ol...@ensta.org> wrote:
>> Actually this problem is not only occurring in the kres profile: I
>> just had a random failure in the BenchmarkTest that uses the full
>> launcher because the dbpedia referenced site got a random error 500
>> from dbpedia.org... even though the BenchmarkTest is using the offline
>> mode explicitly: it seems that the yard is somehow ignoring the
>> offline mode.
> Yes this is an issue I am aware of and it will be addressed by two things:
> (1) Referenced Sites will provide metadata so that callers can ask if
> they can operate offline. This can be used by the
> ReferencedSiteEntityEnhancementEngine to determine if it runs in
> Offline of Online mode.
> (2) I will create a small local index that only includes the entities
> needed for the unit tests. This can than be installed in the
> initialization of the integration tests. This feature is an
> requirement for creating Integration tests of the Entityhub. It will
> also allow to use the ReferencedSiteEntityEnhancementEngine with the
> integration test of the Enhancer.
>
> I hope I can provide both things within the coming two weeks. In the
> meanwhile we can deactivate the ReferencedSiteEntityEnhancementEngine
> by adding a Reference to Offline.
Ok +1
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi
On Sat, Apr 9, 2011 at 10:25 PM, Olivier Grisel
<ol...@ensta.org> wrote:
> Actually this problem is not only occurring in the kres profile: I
> just had a random failure in the BenchmarkTest that uses the full
> launcher because the dbpedia referenced site got a random error 500
> from dbpedia.org... even though the BenchmarkTest is using the offline
> mode explicitly: it seems that the yard is somehow ignoring the
> offline mode.
Yes this is an issue I am aware of and it will be addressed by two things:
(1) Referenced Sites will provide metadata so that callers can ask if
they can operate offline. This can be used by the
ReferencedSiteEntityEnhancementEngine to determine if it runs in
Offline of Online mode.
(2) I will create a small local index that only includes the entities
needed for the unit tests. This can than be installed in the
initialization of the integration tests. This feature is an
requirement for creating Integration tests of the Entityhub. It will
also allow to use the ReferencedSiteEntityEnhancementEngine with the
integration test of the Enhancer.
I hope I can provide both things within the coming two weeks. In the
meanwhile we can deactivate the ReferencedSiteEntityEnhancementEngine
by adding a Reference to Offline.
best
Rupert
>
> --
> Olivier
>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
Actually this problem is not only occurring in the kres profile: I
just had a random failure in the BenchmarkTest that uses the full
launcher because the dbpedia referenced site got a random error 500
from dbpedia.org... even though the BenchmarkTest is using the offline
mode explicitly: it seems that the yard is somehow ignoring the
offline mode.
--
Olivier
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
+1 for all of this :)
--
Olivier
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/11 Bertrand Delacretaz <bd...@apache.org>:
> Hi,
>
> On Mon, Apr 11, 2011 at 6:01 PM, Alessandro Adamou <ad...@cs.unibo.it> wrote:
>> Just a hint but... is there any chance that a build parameter such as:
>>
>> % mvn install -Ponline.tests
>>
>> be passed so that unit tests are performed online only if set to do so?...
>
> We already have this, see commons/stanboltools/offline
>
> It's currently used in the enhancer/integration-tests by passing
> -Dorg.apache.stanbol.offline.mode=true to the build, and some
> components use @Reference to the OnlineMode or OfflineMode services to
> disable themselves. We can easily provide different ways of using
> these settings.
FYI I moved them upstairs into: stanbol/integration-tests directly.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Mon, Apr 11, 2011 at 6:01 PM, Alessandro Adamou <ad...@cs.unibo.it> wrote:
> Just a hint but... is there any chance that a build parameter such as:
>
> % mvn install -Ponline.tests
>
> be passed so that unit tests are performed online only if set to do so?...
We already have this, see commons/stanboltools/offline
It's currently used in the enhancer/integration-tests by passing
-Dorg.apache.stanbol.offline.mode=true to the build, and some
components use @Reference to the OnlineMode or OfflineMode services to
disable themselves. We can easily provide different ways of using
these settings.
-Bertrand
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
> Just a hint but... is there any chance that a build parameter such as:
>
> % mvn install -Ponline.tests
>
> be passed so that unit tests are performed online only if set to do so?
>
> I guess some custom Maven plugin would be required for managing this param.
> Could it be worth it?
Yes sure, but it should not be the default. Otherwise you can write
"online" tests that are excluded from the maven sure-fire build as
explained earlier but that can be run manually from inside eclipse for
instance.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Alessandro Adamou <ad...@cs.unibo.it>.
Just a hint but... is there any chance that a build parameter such as:
% mvn install -Ponline.tests
be passed so that unit tests are performed online only if set to do so?
I guess some custom Maven plugin would be required for managing this
param. Could it be worth it?
That said, the rest of what I've read so far is sensible and +1able
Alessandro
On 4/11/11 4:37 PM, Enrico Daga wrote:
> In general we must support an offline mode handling IRI resolutions
> from the OWLApi with our own strategy, depending on the main Stanbol
> mode (Online/Offline). This can be done by implementing the
> org.semanticweb.owlapi.model.OWLOntologyIRIMapper.
> We should also have a look to implementations such:
> * org.semanticweb.owlapi.util.AutoIRIMapper -> which looks for
> mappings from a specific folder (.../resources for instance)
> * org.semanticweb.owlapi.util.CommonBaseIRIMapper
>
> Finally, if the Administrator decide he doesn't want to resolve IRIs
> on the web, it also must expect that missing imports will come and so
> he could have uncomplete ontologies as well.
> At the same time we should probably support some local registry of
> ontologies to be accessed to dinamically load resources when needed.
>
>> Wrt to tests, I think you can/have to get rid of network dependencies. You
>> can use stubs/drivers for simulating valid and invalid testings. Makes
>> sense?
> Yes, indeed. Even if I think we should be able to test our code
> without the need of HTTP connections, at least in Unit tests
> (integration tests maybe should do that, at least to test the REST
> services).
>
>> I hope this helps.
>> Val
>>
>>
>> On Apr 11, 2011, at 3:09 PM, Olivier Grisel wrote:
>>
>> 2011/4/11 Enrico Daga<en...@gmail.com>:
>>
>> On 11 April 2011 14:37, Olivier Grisel<ol...@ensta.org> wrote:
>>
>> 2011/4/11 Alessandro Adamou<ad...@cs.unibo.it>:
>>
>> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>>
>> The Apache Sling Installer framework could be used to install the
>>
>> needed Ontologies, Rules ... e.g. to the persistent store or some
>>
>> other component.
>>
>> Such resources can be included in the launcher or also in bundles.
>>
>> The only thing one need to do is to implement an installer that
>>
>> recognizes and processes such resources. As an example you could have
>>
>> a look at [1] that is an implementation that processes solr index
>>
>> archives and installs them to the EmbeddedSolrServer used by the
>>
>> SolrYard.
>>
>> We'll look into that, however I cannot tell if using the
>>
>> ontologymanager/store can be of help for another related issue.
>>
>> Import statements in ontologies (and by extension in recipes).
>>
>> These should always be absolute URIs, when you spot relative imports they're
>>
>> non-canon and wrong.
>>
>> Obviously we cannot construct absolute file:// URIs that work for everyone,
>>
>> so they gotta be http URIs, no way out of that.
>>
>> A possible startegy could be that everytime an import is about to be
>>
>> resolved, we first try to hijack to a local resource (e.g. import
>>
>> "http://www.example.org/blahblahblah/myontology" could first try to resolve
>>
>> to "src/main/resources/myontology" or "src/main/resources/myontology", to
>>
>> Yes, or using the ontology store or the DataFileProvider service
>>
>> contributed by Bertrand.
>>
>> then go online if both fail).
>>
>> No, if the engine is in offline mode it should just lookup the local
>>
>> resources and not try to silently use the network in case the resource
>>
>> is not available. It should just raise an IOException with an explicit
>>
>> error message for instance. We need a deterministic offline mode, not
>>
>> just for the tests but also in production: potential users might want
>>
>> to run Stanbol on their IT infrastructure on servers that do not have
>>
>> access to the Internet.
>>
>> As temporary solution I think we could do in this way, at least on tests.
>>
>> The problem - in general - is that the 'owl:import' statement implies
>>
>> the ontology to be fetched/retrieved from the given location IRI and
>>
>> then it's content considered to be part of the ontology.
>>
>> If we want to support an offline mode for ontology-related
>>
>> functionalities (which I see it is good), we should integrate
>>
>> something like:
>>
>> - Local repo of ontologies to lookup (maybe relying on the
>>
>> DataFileProvider service?)
>>
>> - Definition of our own implementation of
>>
>> org.semanticweb.owlapi.OWLOntologyIRIMapper to setup through
>>
>> owlOntologyManager.addIRIMapper(OWLOntologyIRIMapper mapper);
>>
>> This problem, as Alessandro says, is common to the Store module, which
>>
>> also needs to compute owl:import statements.
>>
>> We should find a common solution.
>>
>> +1 for a custom OWLOntologyIRIMapper integrated with the stanbol OSGi
>> runtime.
>>
>> --
>> Olivier
>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>
>>
>> ------------------------------------------------------------
>> Valentina Presutti
>> Semantic Technology Laboratory (STLab)
>> Institute for Cognitive Science and Technology (ISTC)
>> National Research Council (CNR)
>> Via Nomentana 56, Rome - Italy
>> icq# 122838754
>> msn vpresutti@hotmail.it
>> skype bluvale
>>
>
>
Re: About the test suite in the kres profile
Posted by Enrico Daga <en...@gmail.com>.
On 11 April 2011 15:55, valentina presutti <va...@cnr.it> wrote:
> Tools like TopBraid Composer are able to work offline when an import cannot
> be resolved through a local repository.
> They just keep the external references to the entities used in the local
> ontology in a ghost-way i.e. you know/see their URI but not their formal
> definitions stated in the imported ontology.
> Of course you cannot benefit from the axioms defined in the imported
> ontology, you don't have it! But this is perfectly fine, I think, if you
> don't have a local copy.
> Wouldn't his be a policy applicable to ontonet too?
Exactly.
In general we must support an offline mode handling IRI resolutions
from the OWLApi with our own strategy, depending on the main Stanbol
mode (Online/Offline). This can be done by implementing the
org.semanticweb.owlapi.model.OWLOntologyIRIMapper.
We should also have a look to implementations such:
* org.semanticweb.owlapi.util.AutoIRIMapper -> which looks for
mappings from a specific folder (.../resources for instance)
* org.semanticweb.owlapi.util.CommonBaseIRIMapper
Finally, if the Administrator decide he doesn't want to resolve IRIs
on the web, it also must expect that missing imports will come and so
he could have uncomplete ontologies as well.
At the same time we should probably support some local registry of
ontologies to be accessed to dinamically load resources when needed.
> Wrt to tests, I think you can/have to get rid of network dependencies. You
> can use stubs/drivers for simulating valid and invalid testings. Makes
> sense?
Yes, indeed. Even if I think we should be able to test our code
without the need of HTTP connections, at least in Unit tests
(integration tests maybe should do that, at least to test the REST
services).
> I hope this helps.
> Val
>
>
> On Apr 11, 2011, at 3:09 PM, Olivier Grisel wrote:
>
> 2011/4/11 Enrico Daga <en...@gmail.com>:
>
> On 11 April 2011 14:37, Olivier Grisel <ol...@ensta.org> wrote:
>
> 2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
>
> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>
> The Apache Sling Installer framework could be used to install the
>
> needed Ontologies, Rules ... e.g. to the persistent store or some
>
> other component.
>
> Such resources can be included in the launcher or also in bundles.
>
> The only thing one need to do is to implement an installer that
>
> recognizes and processes such resources. As an example you could have
>
> a look at [1] that is an implementation that processes solr index
>
> archives and installs them to the EmbeddedSolrServer used by the
>
> SolrYard.
>
> We'll look into that, however I cannot tell if using the
>
> ontologymanager/store can be of help for another related issue.
>
> Import statements in ontologies (and by extension in recipes).
>
> These should always be absolute URIs, when you spot relative imports they're
>
> non-canon and wrong.
>
> Obviously we cannot construct absolute file:// URIs that work for everyone,
>
> so they gotta be http URIs, no way out of that.
>
> A possible startegy could be that everytime an import is about to be
>
> resolved, we first try to hijack to a local resource (e.g. import
>
> "http://www.example.org/blahblahblah/myontology" could first try to resolve
>
> to "src/main/resources/myontology" or "src/main/resources/myontology", to
>
> Yes, or using the ontology store or the DataFileProvider service
>
> contributed by Bertrand.
>
> then go online if both fail).
>
> No, if the engine is in offline mode it should just lookup the local
>
> resources and not try to silently use the network in case the resource
>
> is not available. It should just raise an IOException with an explicit
>
> error message for instance. We need a deterministic offline mode, not
>
> just for the tests but also in production: potential users might want
>
> to run Stanbol on their IT infrastructure on servers that do not have
>
> access to the Internet.
>
> As temporary solution I think we could do in this way, at least on tests.
>
> The problem - in general - is that the 'owl:import' statement implies
>
> the ontology to be fetched/retrieved from the given location IRI and
>
> then it's content considered to be part of the ontology.
>
> If we want to support an offline mode for ontology-related
>
> functionalities (which I see it is good), we should integrate
>
> something like:
>
> - Local repo of ontologies to lookup (maybe relying on the
>
> DataFileProvider service?)
>
> - Definition of our own implementation of
>
> org.semanticweb.owlapi.OWLOntologyIRIMapper to setup through
>
> owlOntologyManager.addIRIMapper(OWLOntologyIRIMapper mapper);
>
> This problem, as Alessandro says, is common to the Store module, which
>
> also needs to compute owl:import statements.
>
> We should find a common solution.
>
> +1 for a custom OWLOntologyIRIMapper integrated with the stanbol OSGi
> runtime.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
>
> ------------------------------------------------------------
> Valentina Presutti
> Semantic Technology Laboratory (STLab)
> Institute for Cognitive Science and Technology (ISTC)
> National Research Council (CNR)
> Via Nomentana 56, Rome - Italy
> icq# 122838754
> msn vpresutti@hotmail.it
> skype bluvale
>
--
Enrico Daga
--
http://www.enridaga.net
skype: enri-pan
Re: About the test suite in the kres profile
Posted by valentina presutti <va...@cnr.it>.
Tools like TopBraid Composer are able to work offline when an import cannot be resolved through a local repository.
They just keep the external references to the entities used in the local ontology in a ghost-way i.e. you know/see their URI but not their formal definitions stated in the imported ontology.
Of course you cannot benefit from the axioms defined in the imported ontology, you don't have it! But this is perfectly fine, I think, if you don't have a local copy.
Wouldn't his be a policy applicable to ontonet too?
Wrt to tests, I think you can/have to get rid of network dependencies. You can use stubs/drivers for simulating valid and invalid testings. Makes sense?
I hope this helps.
Val
On Apr 11, 2011, at 3:09 PM, Olivier Grisel wrote:
> 2011/4/11 Enrico Daga <en...@gmail.com>:
>> On 11 April 2011 14:37, Olivier Grisel <ol...@ensta.org> wrote:
>>> 2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
>>>> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>>>>>
>>>>> The Apache Sling Installer framework could be used to install the
>>>>> needed Ontologies, Rules ... e.g. to the persistent store or some
>>>>> other component.
>>>>> Such resources can be included in the launcher or also in bundles.
>>>>> The only thing one need to do is to implement an installer that
>>>>> recognizes and processes such resources. As an example you could have
>>>>> a look at [1] that is an implementation that processes solr index
>>>>> archives and installs them to the EmbeddedSolrServer used by the
>>>>> SolrYard.
>>>>
>>>> We'll look into that, however I cannot tell if using the
>>>> ontologymanager/store can be of help for another related issue.
>>>>
>>>> Import statements in ontologies (and by extension in recipes).
>>>>
>>>> These should always be absolute URIs, when you spot relative imports they're
>>>> non-canon and wrong.
>>>>
>>>> Obviously we cannot construct absolute file:// URIs that work for everyone,
>>>> so they gotta be http URIs, no way out of that.
>>>>
>>>> A possible startegy could be that everytime an import is about to be
>>>> resolved, we first try to hijack to a local resource (e.g. import
>>>> "http://www.example.org/blahblahblah/myontology" could first try to resolve
>>>> to "src/main/resources/myontology" or "src/main/resources/myontology", to
>>>
>>> Yes, or using the ontology store or the DataFileProvider service
>>> contributed by Bertrand.
>>>
>>>> then go online if both fail).
>>>
>>> No, if the engine is in offline mode it should just lookup the local
>>> resources and not try to silently use the network in case the resource
>>> is not available. It should just raise an IOException with an explicit
>>> error message for instance. We need a deterministic offline mode, not
>>> just for the tests but also in production: potential users might want
>>> to run Stanbol on their IT infrastructure on servers that do not have
>>> access to the Internet.
>> As temporary solution I think we could do in this way, at least on tests.
>> The problem - in general - is that the 'owl:import' statement implies
>> the ontology to be fetched/retrieved from the given location IRI and
>> then it's content considered to be part of the ontology.
>> If we want to support an offline mode for ontology-related
>> functionalities (which I see it is good), we should integrate
>> something like:
>> - Local repo of ontologies to lookup (maybe relying on the
>> DataFileProvider service?)
>> - Definition of our own implementation of
>> org.semanticweb.owlapi.OWLOntologyIRIMapper to setup through
>> owlOntologyManager.addIRIMapper(OWLOntologyIRIMapper mapper);
>>
>> This problem, as Alessandro says, is common to the Store module, which
>> also needs to compute owl:import statements.
>> We should find a common solution.
>
> +1 for a custom OWLOntologyIRIMapper integrated with the stanbol OSGi runtime.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
------------------------------------------------------------
Valentina Presutti
Semantic Technology Laboratory (STLab)
Institute for Cognitive Science and Technology (ISTC)
National Research Council (CNR)
Via Nomentana 56, Rome - Italy
icq# 122838754
msn vpresutti@hotmail.it
skype bluvale
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/11 Enrico Daga <en...@gmail.com>:
> On 11 April 2011 14:37, Olivier Grisel <ol...@ensta.org> wrote:
>> 2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
>>> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>>>>
>>>> The Apache Sling Installer framework could be used to install the
>>>> needed Ontologies, Rules ... e.g. to the persistent store or some
>>>> other component.
>>>> Such resources can be included in the launcher or also in bundles.
>>>> The only thing one need to do is to implement an installer that
>>>> recognizes and processes such resources. As an example you could have
>>>> a look at [1] that is an implementation that processes solr index
>>>> archives and installs them to the EmbeddedSolrServer used by the
>>>> SolrYard.
>>>
>>> We'll look into that, however I cannot tell if using the
>>> ontologymanager/store can be of help for another related issue.
>>>
>>> Import statements in ontologies (and by extension in recipes).
>>>
>>> These should always be absolute URIs, when you spot relative imports they're
>>> non-canon and wrong.
>>>
>>> Obviously we cannot construct absolute file:// URIs that work for everyone,
>>> so they gotta be http URIs, no way out of that.
>>>
>>> A possible startegy could be that everytime an import is about to be
>>> resolved, we first try to hijack to a local resource (e.g. import
>>> "http://www.example.org/blahblahblah/myontology" could first try to resolve
>>> to "src/main/resources/myontology" or "src/main/resources/myontology", to
>>
>> Yes, or using the ontology store or the DataFileProvider service
>> contributed by Bertrand.
>>
>>> then go online if both fail).
>>
>> No, if the engine is in offline mode it should just lookup the local
>> resources and not try to silently use the network in case the resource
>> is not available. It should just raise an IOException with an explicit
>> error message for instance. We need a deterministic offline mode, not
>> just for the tests but also in production: potential users might want
>> to run Stanbol on their IT infrastructure on servers that do not have
>> access to the Internet.
> As temporary solution I think we could do in this way, at least on tests.
> The problem - in general - is that the 'owl:import' statement implies
> the ontology to be fetched/retrieved from the given location IRI and
> then it's content considered to be part of the ontology.
> If we want to support an offline mode for ontology-related
> functionalities (which I see it is good), we should integrate
> something like:
> - Local repo of ontologies to lookup (maybe relying on the
> DataFileProvider service?)
> - Definition of our own implementation of
> org.semanticweb.owlapi.OWLOntologyIRIMapper to setup through
> owlOntologyManager.addIRIMapper(OWLOntologyIRIMapper mapper);
>
> This problem, as Alessandro says, is common to the Store module, which
> also needs to compute owl:import statements.
> We should find a common solution.
+1 for a custom OWLOntologyIRIMapper integrated with the stanbol OSGi runtime.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Enrico Daga <en...@gmail.com>.
On 11 April 2011 14:37, Olivier Grisel <ol...@ensta.org> wrote:
> 2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
>> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>>>
>>> The Apache Sling Installer framework could be used to install the
>>> needed Ontologies, Rules ... e.g. to the persistent store or some
>>> other component.
>>> Such resources can be included in the launcher or also in bundles.
>>> The only thing one need to do is to implement an installer that
>>> recognizes and processes such resources. As an example you could have
>>> a look at [1] that is an implementation that processes solr index
>>> archives and installs them to the EmbeddedSolrServer used by the
>>> SolrYard.
>>
>> We'll look into that, however I cannot tell if using the
>> ontologymanager/store can be of help for another related issue.
>>
>> Import statements in ontologies (and by extension in recipes).
>>
>> These should always be absolute URIs, when you spot relative imports they're
>> non-canon and wrong.
>>
>> Obviously we cannot construct absolute file:// URIs that work for everyone,
>> so they gotta be http URIs, no way out of that.
>>
>> A possible startegy could be that everytime an import is about to be
>> resolved, we first try to hijack to a local resource (e.g. import
>> "http://www.example.org/blahblahblah/myontology" could first try to resolve
>> to "src/main/resources/myontology" or "src/main/resources/myontology", to
>
> Yes, or using the ontology store or the DataFileProvider service
> contributed by Bertrand.
>
>> then go online if both fail).
>
> No, if the engine is in offline mode it should just lookup the local
> resources and not try to silently use the network in case the resource
> is not available. It should just raise an IOException with an explicit
> error message for instance. We need a deterministic offline mode, not
> just for the tests but also in production: potential users might want
> to run Stanbol on their IT infrastructure on servers that do not have
> access to the Internet.
As temporary solution I think we could do in this way, at least on tests.
The problem - in general - is that the 'owl:import' statement implies
the ontology to be fetched/retrieved from the given location IRI and
then it's content considered to be part of the ontology.
If we want to support an offline mode for ontology-related
functionalities (which I see it is good), we should integrate
something like:
- Local repo of ontologies to lookup (maybe relying on the
DataFileProvider service?)
- Definition of our own implementation of
org.semanticweb.owlapi.OWLOntologyIRIMapper to setup through
owlOntologyManager.addIRIMapper(OWLOntologyIRIMapper mapper);
This problem, as Alessandro says, is common to the Store module, which
also needs to compute owl:import statements.
We should find a common solution.
Enrico
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
--
Enrico Daga
--
http://www.enridaga.net
skype: enri-pan
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
> On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
>>
>> The Apache Sling Installer framework could be used to install the
>> needed Ontologies, Rules ... e.g. to the persistent store or some
>> other component.
>> Such resources can be included in the launcher or also in bundles.
>> The only thing one need to do is to implement an installer that
>> recognizes and processes such resources. As an example you could have
>> a look at [1] that is an implementation that processes solr index
>> archives and installs them to the EmbeddedSolrServer used by the
>> SolrYard.
>
> We'll look into that, however I cannot tell if using the
> ontologymanager/store can be of help for another related issue.
>
> Import statements in ontologies (and by extension in recipes).
>
> These should always be absolute URIs, when you spot relative imports they're
> non-canon and wrong.
>
> Obviously we cannot construct absolute file:// URIs that work for everyone,
> so they gotta be http URIs, no way out of that.
>
> A possible startegy could be that everytime an import is about to be
> resolved, we first try to hijack to a local resource (e.g. import
> "http://www.example.org/blahblahblah/myontology" could first try to resolve
> to "src/main/resources/myontology" or "src/main/resources/myontology", to
Yes, or using the ontology store or the DataFileProvider service
contributed by Bertrand.
> then go online if both fail).
No, if the engine is in offline mode it should just lookup the local
resources and not try to silently use the network in case the resource
is not available. It should just raise an IOException with an explicit
error message for instance. We need a deterministic offline mode, not
just for the tests but also in production: potential users might want
to run Stanbol on their IT infrastructure on servers that do not have
access to the Internet.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Alessandro Adamou <ad...@cs.unibo.it>.
On 4/11/11 2:04 PM, Rupert Westenthaler wrote:
> The Apache Sling Installer framework could be used to install the
> needed Ontologies, Rules ... e.g. to the persistent store or some
> other component.
> Such resources can be included in the launcher or also in bundles.
> The only thing one need to do is to implement an installer that
> recognizes and processes such resources. As an example you could have
> a look at [1] that is an implementation that processes solr index
> archives and installs them to the EmbeddedSolrServer used by the
> SolrYard.
We'll look into that, however I cannot tell if using the
ontologymanager/store can be of help for another related issue.
Import statements in ontologies (and by extension in recipes).
These should always be absolute URIs, when you spot relative imports
they're non-canon and wrong.
Obviously we cannot construct absolute file:// URIs that work for
everyone, so they gotta be http URIs, no way out of that.
A possible startegy could be that everytime an import is about to be
resolved, we first try to hijack to a local resource (e.g. import
"http://www.example.org/blahblahblah/myontology" could first try to
resolve to "src/main/resources/myontology" or
"src/main/resources/myontology", to then go online if both fail).
Bit of like what protégé 4 does, but that's quite a drastic import
resolution policy to be agreed upon.
Alessandro
Re: About the test suite in the kres profile
Posted by Rupert Westenthaler <ru...@gmail.com>.
On Mon, Apr 11, 2011 at 10:55 AM, Alessandro Adamou <ad...@cs.unibo.it> wrote:
> Hi Olivier,
>
>> I am running the tests with "mvn clean install -Pkres" and I have too
>> issues (most likely related):
>>
>> - they need internet access: this means they will randomly fail if the
>> remote resources are not available (network is down, remote server is
>> down, ...)
>> - some of them are much too slow: to ensuite people will regularly run
>> the tests before checking in changes into the
>
> Right you are. Problem is, we cannot live without ensuring that HTTP can be
> resolved at all times
>
> We were thinking of adopting this strategy: unit tests try to fetch remote
> ontologies; if they fail, they fall back to the corresponding local
> resources. This would keep us safe wrt failures, but not wrt test duration
> (suppose we have to wait for timeouts in order to fall back).
>
The Apache Sling Installer framework could be used to install the
needed Ontologies, Rules ... e.g. to the persistent store or some
other component.
Such resources can be included in the launcher or also in bundles.
The only thing one need to do is to implement an installer that
recognizes and processes such resources. As an example you could have
a look at [1] that is an implementation that processes solr index
archives and installs them to the EmbeddedSolrServer used by the
SolrYard.
best
Rupert Westenthaler
[1] http://svn.apache.org/repos/asf/incubator/stanbol/trunk/entityhub/yard/solr/src/main/java/org/apache/stanbol/entityhub/yard/solr/impl/install/
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: About the test suite in the kres profile
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Mon, Apr 11, 2011 at 11:12 AM, Olivier Grisel
<ol...@ensta.org> wrote:
> 2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
>> ...We were thinking of adopting this strategy: unit tests try to fetch remote
>> ontologies; if they fail, they fall back to the corresponding local
>> resources. This would keep us safe wrt failures, but not wrt test duration
>> (suppose we have to wait for timeouts in order to fall back).
>
> I would rather have an explicit "offline" API rather that random
> guessing depending on the state of the network....
Note that the commons/stanboltools/offline module can help for that.
-Bertrand
Re: About the test suite in the kres profile
Posted by Olivier Grisel <ol...@ensta.org>.
2011/4/11 Alessandro Adamou <ad...@cs.unibo.it>:
> Hi Olivier,
>
>> I am running the tests with "mvn clean install -Pkres" and I have too
>> issues (most likely related):
>>
>> - they need internet access: this means they will randomly fail if the
>> remote resources are not available (network is down, remote server is
>> down, ...)
>> - some of them are much too slow: to ensuite people will regularly run
>> the tests before checking in changes into the
>
> Right you are. Problem is, we cannot live without ensuring that HTTP can be
> resolved at all times
But this will probably will have to change in the future: one need to
find a way to make the rule engine and the ontology manager be able to
work only with preloaded resources (either using the entityhub or the
ontology manager own's triple store). One cannot expect a Stanbol
instance to always have a reliable Internet connection.
> We were thinking of adopting this strategy: unit tests try to fetch remote
> ontologies; if they fail, they fall back to the corresponding local
> resources. This would keep us safe wrt failures, but not wrt test duration
> (suppose we have to wait for timeouts in order to fall back).
I would rather have an explicit "offline" API rather that random
guessing depending on the state of the network.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: About the test suite in the kres profile
Posted by Alessandro Adamou <ad...@cs.unibo.it>.
Hi Olivier,
> I am running the tests with "mvn clean install -Pkres" and I have too
> issues (most likely related):
>
> - they need internet access: this means they will randomly fail if the
> remote resources are not available (network is down, remote server is
> down, ...)
> - some of them are much too slow: to ensuite people will regularly run
> the tests before checking in changes into the
Right you are. Problem is, we cannot live without ensuring that HTTP can
be resolved at all times
We were thinking of adopting this strategy: unit tests try to fetch
remote ontologies; if they fail, they fall back to the corresponding
local resources. This would keep us safe wrt failures, but not wrt test
duration (suppose we have to wait for timeouts in order to fall back).