You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Andreas Kuckartz <a....@ping.de> on 2013/10/07 11:30:10 UTC
Update on integration test problems
Rupert Westenthaler:
> Based on the logs i can not see any reason why so much tests do fail:
>
> * most of the warnings and exceptions are expected (because a lot of
> the tests do test illegal requests that are expected to result in
> logged exceptions on the server)
> * java.io.FilePermission on the FST modles of the new Lucene FST
> engine. This might be an issue, but would only cause two tests to
> fail.
> * org.apache.solr.common.SolrException: Timeout waiting for all
> directory ref counts to be released - gave up waiting on - But I think
> this was only when closing down the server after the failed tests.
>
> Can you please
>
> (1) start the stable launcher (launchers/stable/target) and send me
> the startup logs.
Doing that one cause of the problems immediately became clear:
port 8080 was already used x-/
Interestingly there seems to be no entry in error.log indicating _such_
a problem.
But the integration test still fails. Startup log after freeing the port
is appended below. It contains an error "Unresolved constraint in bundle
org.apache.stanbol.enhancer.engines.refactor".
> * if the dbpedia referenced site is present
> (http://localhost:8080/entityhub/site/dbpedia)
> * if the dbpedia solr yard is present
> (http://localhost:8080/solr/default/dbpedia/select?q=*:*)
Yes, both exist running the stable launcher. I used the jar-file without
the "original-" in front of the filename. (What is the difference
between these two?)
> (2) start the full launcher (launchers/full/target) and do not forget
> to set "-XX:MaxPermSize=256M" and check if the two service endpoints
> are available.
Yes, both exist running the full launcher. I used the jar-file without
the "original-" in front of the filename.
> (3) If both endpoints are available for the full launcher please try
> to run the integration tests against this server by going to the
> integration-tests dir in a different cmd window and running: "mvn -o
> test -Dtest.server.url=http://localhost:8080".
Failed tests:
testCRUD(org.apache.stanbol.ontologymanager.web.it.SessionTest): GET
request to http://localhost:8080/ontonet/session: expecting status 200
(content: <html>(..)
testSupportedRDFFormats(org.apache.stanbol.ontologymanager.web.it.SessionTest):
GET request to http://localhost:8080/ontonet/session: expecting status
200 (content: <html>(..)
testSupportedOWLFormats(org.apache.stanbol.ontologymanager.web.it.SessionTest):
GET request to http://localhost:8080/ontonet/session: expecting status
200 (content: <html>(..)
testCRUD(org.apache.stanbol.ontologymanager.web.it.ScopeTest): GET
request to http://localhost:8080/ontonet/ontology: expecting status 200
(content: <html>(..)
testSupportedRDFFormats(org.apache.stanbol.ontologymanager.web.it.ScopeTest):
GET request to http://localhost:8080/ontonet/ontology: expecting status
200 (content: <html>(..)
testActive(org.apache.stanbol.ontologymanager.web.it.ScopeTest): GET
request to http://localhost:8080/ontonet/ontology: expecting status 200
(content: <html>(..)
testSupportedOWLFormats(org.apache.stanbol.ontologymanager.web.it.ScopeTest):
GET request to http://localhost:8080/ontonet/ontology: expecting status
200 (content: <html>(..)
> (4) If this fails please retry the same by including the
> "-no-security" option when starting the full launcher.
Same failed tests.
Cheers,
Andreas
$ java -Xmx1g -jar
launchers/stable/target/org.apache.stanbol.launchers.stable-0.10.0-SNAPSHOT.jar
07.10.2013 09:55:06.957 *INFO * [main] Setting
sling.home=/home/andreas/workspace/stanbol/stanbol (system property
sling.home)
07.10.2013 09:55:06.979 *INFO * [main] Starting Apache Sling in
/home/andreas/workspace/stanbol/stanbol
07.10.2013 09:55:07.207 *INFO * [main] Checking launcher JAR in folder
/home/andreas/workspace/stanbol/stanbol
07.10.2013 09:55:07.438 *INFO * [main] Existing launcher is up to date,
using it: 2.4.0 (org.apache.sling.launchpad.base.jar)
07.10.2013 09:55:07.443 *INFO * [main] Loading launcher class
org.apache.sling.launchpad.base.app.MainDelegate from
org.apache.sling.launchpad.base.jar
07.10.2013 09:55:07.648 *INFO * [main] Setting
sling.launchpad=/home/andreas/workspace/stanbol/stanbol
07.10.2013 09:55:07.649 *INFO * [main] Starting launcher ...
07.10.2013 09:55:07.687 *INFO * [main] HTTP server port: 8080
07.10.2013 09:55:57.937 *INFO* [FelixStartLevel] org.mortbay.log Logging
to org.apache.sling.commons.log.internal.slf4j.SlingLogger@5e1e6cde via
org.mortbay.log.Slf4jLog
INFO : org.apache.felix.scr (95): Version = 1.6.0
[INFO] Started jetty 6.1.x at port(s) HTTP:8080
[INFO] Detected extended HttpService. Filters enabled.
[INFO] Http service whiteboard started
[INFO] Detected extended HttpService. Filters enabled.
[INFO] Started jetty 6.1.x at port(s) HTTP:8080
Okt 07, 2013 9:56:19 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:56:24 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:56:25 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:56:32 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:56:33 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:56:35 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:57:29 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
Okt 07, 2013 9:57:31 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.15 10/30/2012
02:40 PM'
ERROR: Bundle org.apache.stanbol.enhancer.engines.refactor [14]: Error
starting
slinginstall:org.apache.stanbol.enhancer.engines.refactor-0.10.0-SNAPSHOT.jar
(org.osgi.framework.BundleException: Unresolved constraint in bundle
org.apache.stanbol.enhancer.engines.refactor [14]: Unable to resolve
14.0: missing requirement [14.0] package;
(&(package=org.apache.stanbol.ontologymanager.servicesapi.collector)(version>=0.10.0)(!(version>=1.0.0))))
org.osgi.framework.BundleException: Unresolved constraint in bundle
org.apache.stanbol.enhancer.engines.refactor [14]: Unable to resolve
14.0: missing requirement [14.0] package;
(&(package=org.apache.stanbol.ontologymanager.servicesapi.collector)(version>=0.10.0)(!(version>=1.0.0)))
at org.apache.felix.framework.Felix.resolveBundle(Felix.java:3443)
at org.apache.felix.framework.Felix.startBundle(Felix.java:1727)
at
org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1156)
at
org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
at java.lang.Thread.run(Thread.java:724)
07.10.2013 09:57:38.876 *INFO * [main] Startup completed
Re: Jenkins and Java 7 tests Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Rupert Westenthaler:
> There are also nightly builds of Stanbol on dev.iks-project.eu. Those to use
>
> java version "1.7.0_25"
> OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)
> OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> on a Debian box.
And the integration tests do not fail on that machine?
That is almost but not exactly the same version I use:
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Cheers,
Andreas
Re: Jenkins and Java 7 tests Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas
+1 for a Jenkins build on Java 7, but I would also like to keep the
Java 6 build as Java 7 is anyway used by most Stanbol developers.
FYI: Personally I use
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
on Mac OSX
There are also nightly builds of Stanbol on dev.iks-project.eu. Those to use
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
on a Debian box.
best
Rupert
On Thu, Oct 17, 2013 at 11:30 AM, Andreas Kuckartz <a....@ping.de> wrote:
> Andreas Kuckartz:
>> I wondered why Jenkins does not report integration test failures and
>> looked at
>> https://builds.apache.org/computer/
>>
>> The general Linux test machines are running Ubuntu and all of them seem
>> to be using OpenJDK version 1.6.0_27.
>>
>> In other words: So far Stanbol probably is not tested with any Java 7
>> version by Jenkins. I do think that this should be improved.
>>
>> jdk1.7.0_04 should be available on several of the Ubuntu-machines for
>> more than a year now:
>> https://issues.apache.org/jira/browse/INFRA-5343?focusedCommentId=13468324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13468324
>
> The test was configured 2,5 years ago by Betrand (when Jenkins was still
> Hudson):
> https://issues.apache.org/jira/browse/STANBOL-70
>
> Anyone else who already knows how to setup additional automatic tests
> for a Stanbol and Java 7 ?
>
> I have created an issue for that:
> https://issues.apache.org/jira/browse/STANBOL-1180
>
> Cheers,
> Andreas
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Jenkins and Java 7 tests Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Andreas Kuckartz:
> I wondered why Jenkins does not report integration test failures and
> looked at
> https://builds.apache.org/computer/
>
> The general Linux test machines are running Ubuntu and all of them seem
> to be using OpenJDK version 1.6.0_27.
>
> In other words: So far Stanbol probably is not tested with any Java 7
> version by Jenkins. I do think that this should be improved.
>
> jdk1.7.0_04 should be available on several of the Ubuntu-machines for
> more than a year now:
> https://issues.apache.org/jira/browse/INFRA-5343?focusedCommentId=13468324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13468324
The test was configured 2,5 years ago by Betrand (when Jenkins was still
Hudson):
https://issues.apache.org/jira/browse/STANBOL-70
Anyone else who already knows how to setup additional automatic tests
for a Stanbol and Java 7 ?
I have created an issue for that:
https://issues.apache.org/jira/browse/STANBOL-1180
Cheers,
Andreas
Jenkins and Java 7 tests Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
I wondered why Jenkins does not report integration test failures and
looked at
https://builds.apache.org/computer/
The general Linux test machines are running Ubuntu and all of them seem
to be using OpenJDK version 1.6.0_27.
In other words: So far Stanbol probably is not tested with any Java 7
version by Jenkins. I do think that this should be improved.
jdk1.7.0_04 should be available on several of the Ubuntu-machines for
more than a year now:
https://issues.apache.org/jira/browse/INFRA-5343?focusedCommentId=13468324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13468324
Cheers,
Andreas
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Rupert Westenthaler:
> at least we are making progress ...
> ... loading the FST corpus does now work.
:-)
> ... or is something really wrong with the SecurityManager
> configuration of Andreas JVM?
I had a look at my system-wide java.security and java.policy files but
did not notice anything unusual. But I could have overlooked something.
Cheers,
Andreas
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Rupert Westenthaler:
> BTW: The last Jenkins build showed up the same problem [1]. As I
> assume that the configuration of Jenkins was not changed from Java 6
> to Java 7 I have no Idea why ...
>
> https://builds.apache.org/job/stanbol-trunk-1.6/org.apache.stanbol$org.apache.stanbol.integration-tests/1480/testReport/junit/org.apache.stanbol.enhancer.it/FstLinkingTest/testFstLinkingEnhancement/
That is somewhat scary...
I meanwhile tried to build Stanbol using OpenJDK6 and the integration
tests also failed.
Cheers,
Andreas
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Hi Rupert,
the integrations tests still fail, but the number of failures now is
only 51, which is lower than the number before the last changes were made.
> If not you could try to change StanbolTestBase@177 to catch all
> Exceptions and try again.
Will do that later.
There was/is a little inconsistency regarding pom.xml. I had seen these
errors even when I only executed "mvn clean":
[ERROR] The project org.apache.stanbol:stanbol-reactor:1.0.0-SNAPSHOT
(/home/andreas/workspace/stanbol/pom.xml) has 3 errors
[ERROR] Child module
/home/andreas/workspace/stanbol/development/archetypes/statefull-webmodule
of /home/andreas/workspace/stanbol/pom.xml does not exist
[ERROR] Child module
/home/andreas/workspace/stanbol/development/archetypes/stateless-webmodule
of /home/andreas/workspace/stanbol/pom.xml does not exist
[ERROR] Child module
/home/andreas/workspace/stanbol/development/archetypes/enhancement-engine of
/home/andreas/workspace/stanbol/pom.xml does not exist
The directory stanbol/development/archetypes/ does not exist.
To resolve this I removed the three lines in pom.xml locally. "mvn
install" is currently running.
Cheers,
Andreas
---
Rupert Westenthaler:
> Hi all,
>
> Let me provide a short update on this:
>
> I created [STANBOL-1208](https://issues.apache.org/jira/browse/STANBOL-1208)
> describing this.
>
> After updating to the most current Apache Commons HTTP I was able to
> reproduce a state as described by Andreas that tests are executing
> (and failing) before the server is ready.
>
> For me that was caused by the new version of Apache Commons HTTP is
> now trowing a different Exception and therefore the @Before method of
> the StanbolTestBase fails instead of catching the Exception and
> retrying the request. In other words - an unexpected Exception is
> causing the Reetry-Loop to be exits and tests to fail without waiting
> for the Server to be ready.
>
> With [revision 1539963](http://svn.apache.org/r1539963) I fixed this
> for me by catching all ConnectionExceptions instead of
> HttpHostConnectException. @Andreas this might also fix your problem.
> If not you could try to change StanbolTestBase@177 to catch all
> Exceptions and try again.
>
> best
> Rupert
>
>
> On Fri, Oct 18, 2013 at 12:52 PM, Rupert Westenthaler
> <ru...@gmail.com> wrote:
>> Hi Andreas, all
>>
>> Andreas was sending me several mails with log files for further
>> tracing down his problem. With this mail I will try to summarize the
>> current state.
>>
>> This mail discusses logs in the error.log file after running the
>> integration-tests. This file can be found at
>>
>> integration-tests/target/launchdir/stanbol/logs/error.log
>>
>> and is about 420MByte after a full run of all tests.
>>
>> As for Andreas tests of the Entityhub ReferencedSite and SiteManager
>> Tests are failing the following analysis does focus on those
>> components
>>
>> The following log entries show that the Referenced Site for DBpedia is started:
>>
>> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
>> update ReferencedSite service:
>> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
>> validate available services:
>> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent ...
>> all required Services present.
>> 17.10.2013 23:22:56.069 *INFO* [Thread-47]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
>> register ReferencedSite 'dbpedia'
>>
>> and the Site is also registered with the SiteManager
>>
>> 17.10.2013 23:22:56.246 *DEBUG* [Thread-47]
>> org.apache.stanbol.entityhub.core.impl.SiteManagerImpl ... binding
>> ReferencedSite dbpedia
>>
>> Also unregistration after the tests is working as expected
>>
>> 17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
>> update ReferencedSite service:
>> 17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
>> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
>> unregister ReferencedSite 'dbpedia'
>>
>> That indicates that on the Component level everything works fine!
>>
>> On the RESTful services level we need to have a look at the JAX-RS
>> resources handling the requests sent by the integration test
>>
>> Here we see a lot of entries such as
>>
>> 17.10.2013 23:22:52.705 *DEBUG* [483542560@qtp-909268873-7]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> <init> with site dbpedia
>> 17.10.2013 23:22:52.705 *ERROR* [483542560@qtp-909268873-7]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> Site dbpedia not found (no referenced site with that ID is present
>> within the Entityhub
>>
>> Those ERRORs are expected as the 'dbpedia' site is only registered at
>> timecode '23:22:56.069'. As expected Requests sent after that are
>> correctly handled.
>>
>> 17.10.2013 23:24:51.967 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.commons.security.auth.AuthenticatingFilter
>> filtering request
>> 17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.commons.httpqueryheaders.OverwriteableHeaderHttpServletRequest
>> Accept Header [text/rdf+nt]=null (was '{}')
>> 17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> <init> with site dbpedia
>> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> site/dbpedia/entity Request
>> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> > id : http://dbpedia.org/resource/Paris
>> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> > accept : [text/rdf+nt]
>> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> > mediaType: null
>> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
>> handle Request for Entity http://dbpedia.org/resource/Paris of Site
>> dbpedia
>> 17.10.2013 23:24:51.972 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.yard.solr.impl.SolrYard Create
>> Representation http://dbpedia.org/resource/Paris from SolrDocument
>> 17.10.2013 23:24:51.973 *DEBUG* [1540499998@qtp-909268873-11]
>> org.apache.stanbol.entityhub.yard.solr.impl.SolrYard ... done
>> [retrieve=2ms|create=1ms|sum=3ms]
>>
>>
>> Still the above Errors do indicate a problem:
>>
>> The Integration-Test is supposed to wait until the Server is up and
>> running before sending Requests. In the case of the Integration-Tests
>> for the Entityhub this is ensured by
>>
>> 1. the EntityhubTestBase querying the SiteManager endpoint
>> (entityhub/sites) for the available sites
>> 2. the extending tests to provide a list of required Sites - in case
>> of the ReferencedSiteTest 'dbpedia'
>>
>> However note that there are NO requests sent to the referenced site
>> endpoint. So why does the log show
>>
>> *ERROR* [..] ReferencedSiteRootResource Site dbpedia not found [..]
>>
>> This can only happen if the Integration-Test starts with the test
>> without waiting for the Site to become available. What is most likely
>> caused by a bug in the test itself.
>>
>> I have also an idea why only Andreas is affected by this. I compared
>> the logs of Andreas with those of my local machine and discovered that
>> the tests are executed in a different oder. While in my environment
>> all the Stanbol Enhancer tests where executed first and gave the
>> Entityhub enough time for initialization. With Andreas environment the
>> Entityhub tests where first executed what caused them to fail.
>>
>> On a side note: Also the Enhancer tests do depend on the dbpedia
>> ReferencedSite. So most likely the check for availability of the
>> Entityhub does work for those tests otherwise I should run in similar
>> issues causing Enhancer tests to fail.
>>
>> Will keep you updated
>> best
>> Rupert
>>
>> --
>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>> | Bodenlehenstraße 11 ++43-699-11108907
>> | A-5500 Bischofshofen
>
>
>
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi all,
Let me provide a short update on this:
I created [STANBOL-1208](https://issues.apache.org/jira/browse/STANBOL-1208)
describing this.
After updating to the most current Apache Commons HTTP I was able to
reproduce a state as described by Andreas that tests are executing
(and failing) before the server is ready.
For me that was caused by the new version of Apache Commons HTTP is
now trowing a different Exception and therefore the @Before method of
the StanbolTestBase fails instead of catching the Exception and
retrying the request. In other words - an unexpected Exception is
causing the Reetry-Loop to be exits and tests to fail without waiting
for the Server to be ready.
With [revision 1539963](http://svn.apache.org/r1539963) I fixed this
for me by catching all ConnectionExceptions instead of
HttpHostConnectException. @Andreas this might also fix your problem.
If not you could try to change StanbolTestBase@177 to catch all
Exceptions and try again.
best
Rupert
On Fri, Oct 18, 2013 at 12:52 PM, Rupert Westenthaler
<ru...@gmail.com> wrote:
> Hi Andreas, all
>
> Andreas was sending me several mails with log files for further
> tracing down his problem. With this mail I will try to summarize the
> current state.
>
> This mail discusses logs in the error.log file after running the
> integration-tests. This file can be found at
>
> integration-tests/target/launchdir/stanbol/logs/error.log
>
> and is about 420MByte after a full run of all tests.
>
> As for Andreas tests of the Entityhub ReferencedSite and SiteManager
> Tests are failing the following analysis does focus on those
> components
>
> The following log entries show that the Referenced Site for DBpedia is started:
>
> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
> update ReferencedSite service:
> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
> validate available services:
> 17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent ...
> all required Services present.
> 17.10.2013 23:22:56.069 *INFO* [Thread-47]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
> register ReferencedSite 'dbpedia'
>
> and the Site is also registered with the SiteManager
>
> 17.10.2013 23:22:56.246 *DEBUG* [Thread-47]
> org.apache.stanbol.entityhub.core.impl.SiteManagerImpl ... binding
> ReferencedSite dbpedia
>
> Also unregistration after the tests is working as expected
>
> 17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
> update ReferencedSite service:
> 17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
> org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
> unregister ReferencedSite 'dbpedia'
>
> That indicates that on the Component level everything works fine!
>
> On the RESTful services level we need to have a look at the JAX-RS
> resources handling the requests sent by the integration test
>
> Here we see a lot of entries such as
>
> 17.10.2013 23:22:52.705 *DEBUG* [483542560@qtp-909268873-7]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> <init> with site dbpedia
> 17.10.2013 23:22:52.705 *ERROR* [483542560@qtp-909268873-7]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> Site dbpedia not found (no referenced site with that ID is present
> within the Entityhub
>
> Those ERRORs are expected as the 'dbpedia' site is only registered at
> timecode '23:22:56.069'. As expected Requests sent after that are
> correctly handled.
>
> 17.10.2013 23:24:51.967 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.commons.security.auth.AuthenticatingFilter
> filtering request
> 17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.commons.httpqueryheaders.OverwriteableHeaderHttpServletRequest
> Accept Header [text/rdf+nt]=null (was '{}')
> 17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> <init> with site dbpedia
> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> site/dbpedia/entity Request
> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> > id : http://dbpedia.org/resource/Paris
> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> > accept : [text/rdf+nt]
> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> > mediaType: null
> 17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> handle Request for Entity http://dbpedia.org/resource/Paris of Site
> dbpedia
> 17.10.2013 23:24:51.972 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.yard.solr.impl.SolrYard Create
> Representation http://dbpedia.org/resource/Paris from SolrDocument
> 17.10.2013 23:24:51.973 *DEBUG* [1540499998@qtp-909268873-11]
> org.apache.stanbol.entityhub.yard.solr.impl.SolrYard ... done
> [retrieve=2ms|create=1ms|sum=3ms]
>
>
> Still the above Errors do indicate a problem:
>
> The Integration-Test is supposed to wait until the Server is up and
> running before sending Requests. In the case of the Integration-Tests
> for the Entityhub this is ensured by
>
> 1. the EntityhubTestBase querying the SiteManager endpoint
> (entityhub/sites) for the available sites
> 2. the extending tests to provide a list of required Sites - in case
> of the ReferencedSiteTest 'dbpedia'
>
> However note that there are NO requests sent to the referenced site
> endpoint. So why does the log show
>
> *ERROR* [..] ReferencedSiteRootResource Site dbpedia not found [..]
>
> This can only happen if the Integration-Test starts with the test
> without waiting for the Site to become available. What is most likely
> caused by a bug in the test itself.
>
> I have also an idea why only Andreas is affected by this. I compared
> the logs of Andreas with those of my local machine and discovered that
> the tests are executed in a different oder. While in my environment
> all the Stanbol Enhancer tests where executed first and gave the
> Entityhub enough time for initialization. With Andreas environment the
> Entityhub tests where first executed what caused them to fail.
>
> On a side note: Also the Enhancer tests do depend on the dbpedia
> ReferencedSite. So most likely the check for availability of the
> Entityhub does work for those tests otherwise I should run in similar
> issues causing Enhancer tests to fail.
>
> Will keep you updated
> best
> Rupert
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas, all
Andreas was sending me several mails with log files for further
tracing down his problem. With this mail I will try to summarize the
current state.
This mail discusses logs in the error.log file after running the
integration-tests. This file can be found at
integration-tests/target/launchdir/stanbol/logs/error.log
and is about 420MByte after a full run of all tests.
As for Andreas tests of the Entityhub ReferencedSite and SiteManager
Tests are failing the following analysis does focus on those
components
The following log entries show that the Referenced Site for DBpedia is started:
17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
update ReferencedSite service:
17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
validate available services:
17.10.2013 23:22:56.069 *DEBUG* [Thread-47]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent ...
all required Services present.
17.10.2013 23:22:56.069 *INFO* [Thread-47]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
register ReferencedSite 'dbpedia'
and the Site is also registered with the SiteManager
17.10.2013 23:22:56.246 *DEBUG* [Thread-47]
org.apache.stanbol.entityhub.core.impl.SiteManagerImpl ... binding
ReferencedSite dbpedia
Also unregistration after the tests is working as expected
17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent >
update ReferencedSite service:
17.10.2013 23:29:51.639 *DEBUG* [FelixStartLevel]
org.apache.stanbol.entityhub.core.impl.ReferencedSiteComponent -
unregister ReferencedSite 'dbpedia'
That indicates that on the Component level everything works fine!
On the RESTful services level we need to have a look at the JAX-RS
resources handling the requests sent by the integration test
Here we see a lot of entries such as
17.10.2013 23:22:52.705 *DEBUG* [483542560@qtp-909268873-7]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
<init> with site dbpedia
17.10.2013 23:22:52.705 *ERROR* [483542560@qtp-909268873-7]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
Site dbpedia not found (no referenced site with that ID is present
within the Entityhub
Those ERRORs are expected as the 'dbpedia' site is only registered at
timecode '23:22:56.069'. As expected Requests sent after that are
correctly handled.
17.10.2013 23:24:51.967 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.commons.security.auth.AuthenticatingFilter
filtering request
17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.commons.httpqueryheaders.OverwriteableHeaderHttpServletRequest
Accept Header [text/rdf+nt]=null (was '{}')
17.10.2013 23:24:51.968 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
<init> with site dbpedia
17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
site/dbpedia/entity Request
17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> id : http://dbpedia.org/resource/Paris
17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> accept : [text/rdf+nt]
17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
> mediaType: null
17.10.2013 23:24:51.970 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.jersey.resource.ReferencedSiteRootResource
handle Request for Entity http://dbpedia.org/resource/Paris of Site
dbpedia
17.10.2013 23:24:51.972 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.yard.solr.impl.SolrYard Create
Representation http://dbpedia.org/resource/Paris from SolrDocument
17.10.2013 23:24:51.973 *DEBUG* [1540499998@qtp-909268873-11]
org.apache.stanbol.entityhub.yard.solr.impl.SolrYard ... done
[retrieve=2ms|create=1ms|sum=3ms]
Still the above Errors do indicate a problem:
The Integration-Test is supposed to wait until the Server is up and
running before sending Requests. In the case of the Integration-Tests
for the Entityhub this is ensured by
1. the EntityhubTestBase querying the SiteManager endpoint
(entityhub/sites) for the available sites
2. the extending tests to provide a list of required Sites - in case
of the ReferencedSiteTest 'dbpedia'
However note that there are NO requests sent to the referenced site
endpoint. So why does the log show
*ERROR* [..] ReferencedSiteRootResource Site dbpedia not found [..]
This can only happen if the Integration-Test starts with the test
without waiting for the Site to become available. What is most likely
caused by a bug in the test itself.
I have also an idea why only Andreas is affected by this. I compared
the logs of Andreas with those of my local machine and discovered that
the tests are executed in a different oder. While in my environment
all the Stanbol Enhancer tests where executed first and gave the
Entityhub enough time for initialization. With Andreas environment the
Entityhub tests where first executed what caused them to fail.
On a side note: Also the Enhancer tests do depend on the dbpedia
ReferencedSite. So most likely the check for availability of the
Entityhub does work for those tests otherwise I should run in similar
issues causing Enhancer tests to fail.
Will keep you updated
best
Rupert
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas, all
With http://svn.apache.org/r1533055 the Attributes are loaded in a
doPrivileged() block. Can you check if this finally solves the issue
for you.
best
Rupert
On Thu, Oct 17, 2013 at 1:09 PM, Rupert Westenthaler
<ru...@gmail.com> wrote:
> BTW: The last Jenkins build showed up the same problem [1]. As I
> assume that the configuration of Jenkins was not changed from Java 6
> to Java 7 I have no Idea why ...
>
> https://builds.apache.org/job/stanbol-trunk-1.6/org.apache.stanbol$org.apache.stanbol.integration-tests/1480/testReport/junit/org.apache.stanbol.enhancer.it/FstLinkingTest/testFstLinkingEnhancement/
>
> best
> Rupert
>
>
> On Thu, Oct 17, 2013 at 12:59 PM, Rupert Westenthaler
> <ru...@gmail.com> wrote:
>> HI Reto,
>>
>> Thanks for the information. If the 'getClassLoader' permission is not
>> included in the default I will change the engine to load the Solr
>> Attribute in a doPrivileged(..) block. The class name is anyway
>> statically in the code (and not parsed via a configuration). So there
>> is no danger that a user can load remote code.
>>
>> best
>> Rupert
>>
>>
>> On Thu, Oct 17, 2013 at 11:37 AM, Reto Bachmann-Gmür <re...@apache.org> wrote:
>>> Hi Rupert, Andreas, all
>>>
>>> It seems that indeed ("java.lang.RuntimePermission" "getClassLoader") is not
>>> part of the basePermissionRole in the 0.X branch but is part of this role in
>>> the 1.0 branch. The reason is that the ng branch is using
>>> org.apache.clerezza.platform.config:0.4.0.SNAPSHOT.
>>>
>>> The more fundamental issue is that the default system graph is just provided
>>> by one bundle which sets the system graph if this isn't there yet. Bundles
>>> could of course modify the system graph in ntheir activators but there
>>> should be a mechanism that ensures a bundle makes its modifications only
>>> once and doesn't overwrite subsequent user made changes.
>>>
>>> Cheers,
>>> Reto
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Oct 17, 2013 at 6:31 AM, Rupert Westenthaler
>>> <ru...@gmail.com> wrote:
>>>>
>>>> Hi Andreas, Reto, all
>>>>
>>>> at least we are making progress ...
>>>> ... loading the FST corpus does now work.
>>>>
>>>> But now the engine is running into an other security related problem
>>>>
>>>> java.security.AccessControlException: access denied
>>>> ("java.lang.RuntimePermission" "getClassLoader")
>>>> at
>>>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
>>>> at
>>>> java.security.AccessController.checkPermission(AccessController.java:559)
>>>> at
>>>> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>>>> at
>>>> java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1549)
>>>> at java.lang.Class.getClassLoader(Class.java:614)
>>>> at
>>>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:79)
>>>> at
>>>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
>>>> at
>>>> org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
>>>> at
>>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.LinkableTokenFilter.<init>(LinkableTokenFilter.java:148)
>>>> at
>>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.tag(FstLinkingEngine.java:358)
>>>> at
>>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.computeEnhancements(FstLinkingEngine.java:175)
>>>>
>>>> indicating that the SecurityManager denies Solr access to the Classloader.
>>>>
>>>> LinkableTokenFilter.java:148 registers an Solr Analyzer Attribute to
>>>> the TokenFilter. The Attribute in question is provided by a JAR that
>>>> is embedded in the same bundle. Solr uses "Class.forName(..)" in
>>>> AttributeSource.java:79.
>>>>
>>>> @Reto: it it expected that the SecurityManager denies access to the
>>>> Classloader used by Class.forName(..). Should I move the
>>>> initialization of the Attributes in a doPrivileged(..) block, or is
>>>> something really wrong with the SecurityManager configuration of
>>>> Andreas JVM?
>>>>
>>>> best
>>>> Rupert
>>>>
>>>> On Wed, Oct 16, 2013 at 3:31 PM, Andreas Kuckartz <a....@ping.de>
>>>> wrote:
>>>> > Hi Rupert,
>>>> >
>>>> > thanks again, but unfortunately the integration tests still fail. Log
>>>> > file is appended.
>>>> >
>>>> > Cheers,
>>>> > Andreas
>>>> > ---
>>>> >
>>>> > Rupert Westenthaler:
>>>> >> Hi Andreas
>>>> >>
>>>> >> Based on the new error log the problem was accessing the last
>>>> >> modification date of the FST corpus file outside the doPrivileged()
>>>> >> block.
>>>> >> I have fixed this with http://svn.apache.org/r1532742.
>>>> >>
>>>> >> Would be nice if you could check again
>>>> >>
>>>> >> best
>>>> >> Rupert
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>>>> | Bodenlehenstraße 11 ++43-699-11108907
>>>> | A-5500 Bischofshofen
>>>
>>>
>>
>>
>>
>> --
>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>> | Bodenlehenstraße 11 ++43-699-11108907
>> | A-5500 Bischofshofen
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
BTW: The last Jenkins build showed up the same problem [1]. As I
assume that the configuration of Jenkins was not changed from Java 6
to Java 7 I have no Idea why ...
https://builds.apache.org/job/stanbol-trunk-1.6/org.apache.stanbol$org.apache.stanbol.integration-tests/1480/testReport/junit/org.apache.stanbol.enhancer.it/FstLinkingTest/testFstLinkingEnhancement/
best
Rupert
On Thu, Oct 17, 2013 at 12:59 PM, Rupert Westenthaler
<ru...@gmail.com> wrote:
> HI Reto,
>
> Thanks for the information. If the 'getClassLoader' permission is not
> included in the default I will change the engine to load the Solr
> Attribute in a doPrivileged(..) block. The class name is anyway
> statically in the code (and not parsed via a configuration). So there
> is no danger that a user can load remote code.
>
> best
> Rupert
>
>
> On Thu, Oct 17, 2013 at 11:37 AM, Reto Bachmann-Gmür <re...@apache.org> wrote:
>> Hi Rupert, Andreas, all
>>
>> It seems that indeed ("java.lang.RuntimePermission" "getClassLoader") is not
>> part of the basePermissionRole in the 0.X branch but is part of this role in
>> the 1.0 branch. The reason is that the ng branch is using
>> org.apache.clerezza.platform.config:0.4.0.SNAPSHOT.
>>
>> The more fundamental issue is that the default system graph is just provided
>> by one bundle which sets the system graph if this isn't there yet. Bundles
>> could of course modify the system graph in ntheir activators but there
>> should be a mechanism that ensures a bundle makes its modifications only
>> once and doesn't overwrite subsequent user made changes.
>>
>> Cheers,
>> Reto
>>
>>
>>
>>
>>
>>
>> On Thu, Oct 17, 2013 at 6:31 AM, Rupert Westenthaler
>> <ru...@gmail.com> wrote:
>>>
>>> Hi Andreas, Reto, all
>>>
>>> at least we are making progress ...
>>> ... loading the FST corpus does now work.
>>>
>>> But now the engine is running into an other security related problem
>>>
>>> java.security.AccessControlException: access denied
>>> ("java.lang.RuntimePermission" "getClassLoader")
>>> at
>>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
>>> at
>>> java.security.AccessController.checkPermission(AccessController.java:559)
>>> at
>>> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>>> at
>>> java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1549)
>>> at java.lang.Class.getClassLoader(Class.java:614)
>>> at
>>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:79)
>>> at
>>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
>>> at
>>> org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
>>> at
>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.LinkableTokenFilter.<init>(LinkableTokenFilter.java:148)
>>> at
>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.tag(FstLinkingEngine.java:358)
>>> at
>>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.computeEnhancements(FstLinkingEngine.java:175)
>>>
>>> indicating that the SecurityManager denies Solr access to the Classloader.
>>>
>>> LinkableTokenFilter.java:148 registers an Solr Analyzer Attribute to
>>> the TokenFilter. The Attribute in question is provided by a JAR that
>>> is embedded in the same bundle. Solr uses "Class.forName(..)" in
>>> AttributeSource.java:79.
>>>
>>> @Reto: it it expected that the SecurityManager denies access to the
>>> Classloader used by Class.forName(..). Should I move the
>>> initialization of the Attributes in a doPrivileged(..) block, or is
>>> something really wrong with the SecurityManager configuration of
>>> Andreas JVM?
>>>
>>> best
>>> Rupert
>>>
>>> On Wed, Oct 16, 2013 at 3:31 PM, Andreas Kuckartz <a....@ping.de>
>>> wrote:
>>> > Hi Rupert,
>>> >
>>> > thanks again, but unfortunately the integration tests still fail. Log
>>> > file is appended.
>>> >
>>> > Cheers,
>>> > Andreas
>>> > ---
>>> >
>>> > Rupert Westenthaler:
>>> >> Hi Andreas
>>> >>
>>> >> Based on the new error log the problem was accessing the last
>>> >> modification date of the FST corpus file outside the doPrivileged()
>>> >> block.
>>> >> I have fixed this with http://svn.apache.org/r1532742.
>>> >>
>>> >> Would be nice if you could check again
>>> >>
>>> >> best
>>> >> Rupert
>>> >
>>>
>>>
>>>
>>> --
>>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>>> | Bodenlehenstraße 11 ++43-699-11108907
>>> | A-5500 Bischofshofen
>>
>>
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
HI Reto,
Thanks for the information. If the 'getClassLoader' permission is not
included in the default I will change the engine to load the Solr
Attribute in a doPrivileged(..) block. The class name is anyway
statically in the code (and not parsed via a configuration). So there
is no danger that a user can load remote code.
best
Rupert
On Thu, Oct 17, 2013 at 11:37 AM, Reto Bachmann-Gmür <re...@apache.org> wrote:
> Hi Rupert, Andreas, all
>
> It seems that indeed ("java.lang.RuntimePermission" "getClassLoader") is not
> part of the basePermissionRole in the 0.X branch but is part of this role in
> the 1.0 branch. The reason is that the ng branch is using
> org.apache.clerezza.platform.config:0.4.0.SNAPSHOT.
>
> The more fundamental issue is that the default system graph is just provided
> by one bundle which sets the system graph if this isn't there yet. Bundles
> could of course modify the system graph in ntheir activators but there
> should be a mechanism that ensures a bundle makes its modifications only
> once and doesn't overwrite subsequent user made changes.
>
> Cheers,
> Reto
>
>
>
>
>
>
> On Thu, Oct 17, 2013 at 6:31 AM, Rupert Westenthaler
> <ru...@gmail.com> wrote:
>>
>> Hi Andreas, Reto, all
>>
>> at least we are making progress ...
>> ... loading the FST corpus does now work.
>>
>> But now the engine is running into an other security related problem
>>
>> java.security.AccessControlException: access denied
>> ("java.lang.RuntimePermission" "getClassLoader")
>> at
>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
>> at
>> java.security.AccessController.checkPermission(AccessController.java:559)
>> at
>> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>> at
>> java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1549)
>> at java.lang.Class.getClassLoader(Class.java:614)
>> at
>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:79)
>> at
>> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
>> at
>> org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
>> at
>> org.apache.stanbol.enhancer.engines.lucenefstlinking.LinkableTokenFilter.<init>(LinkableTokenFilter.java:148)
>> at
>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.tag(FstLinkingEngine.java:358)
>> at
>> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.computeEnhancements(FstLinkingEngine.java:175)
>>
>> indicating that the SecurityManager denies Solr access to the Classloader.
>>
>> LinkableTokenFilter.java:148 registers an Solr Analyzer Attribute to
>> the TokenFilter. The Attribute in question is provided by a JAR that
>> is embedded in the same bundle. Solr uses "Class.forName(..)" in
>> AttributeSource.java:79.
>>
>> @Reto: it it expected that the SecurityManager denies access to the
>> Classloader used by Class.forName(..). Should I move the
>> initialization of the Attributes in a doPrivileged(..) block, or is
>> something really wrong with the SecurityManager configuration of
>> Andreas JVM?
>>
>> best
>> Rupert
>>
>> On Wed, Oct 16, 2013 at 3:31 PM, Andreas Kuckartz <a....@ping.de>
>> wrote:
>> > Hi Rupert,
>> >
>> > thanks again, but unfortunately the integration tests still fail. Log
>> > file is appended.
>> >
>> > Cheers,
>> > Andreas
>> > ---
>> >
>> > Rupert Westenthaler:
>> >> Hi Andreas
>> >>
>> >> Based on the new error log the problem was accessing the last
>> >> modification date of the FST corpus file outside the doPrivileged()
>> >> block.
>> >> I have fixed this with http://svn.apache.org/r1532742.
>> >>
>> >> Would be nice if you could check again
>> >>
>> >> best
>> >> Rupert
>> >
>>
>>
>>
>> --
>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>> | Bodenlehenstraße 11 ++43-699-11108907
>> | A-5500 Bischofshofen
>
>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Reto Bachmann-Gmür <re...@apache.org>.
Hi Rupert, Andreas, all
It seems that indeed ("java.lang.RuntimePermission" "getClassLoader") is
not part of the basePermissionRole in the 0.X branch but is part of this
role in the 1.0 branch. The reason is that the ng branch is using
org.apache.clerezza.platform.config:0.4.0.SNAPSHOT.
The more fundamental issue is that the default system graph is just
provided by one bundle which sets the system graph if this isn't there yet.
Bundles could of course modify the system graph in ntheir activators but
there should be a mechanism that ensures a bundle makes its modifications
only once and doesn't overwrite subsequent user made changes.
Cheers,
Reto
On Thu, Oct 17, 2013 at 6:31 AM, Rupert Westenthaler <
rupert.westenthaler@gmail.com> wrote:
> Hi Andreas, Reto, all
>
> at least we are making progress ...
> ... loading the FST corpus does now work.
>
> But now the engine is running into an other security related problem
>
> java.security.AccessControlException: access denied
> ("java.lang.RuntimePermission" "getClassLoader")
> at
> java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
> at
> java.security.AccessController.checkPermission(AccessController.java:559)
> at
> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
> at
> java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1549)
> at java.lang.Class.getClassLoader(Class.java:614)
> at
> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:79)
> at
> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
> at
> org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
> at
> org.apache.stanbol.enhancer.engines.lucenefstlinking.LinkableTokenFilter.<init>(LinkableTokenFilter.java:148)
> at
> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.tag(FstLinkingEngine.java:358)
> at
> org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.computeEnhancements(FstLinkingEngine.java:175)
>
> indicating that the SecurityManager denies Solr access to the Classloader.
>
> LinkableTokenFilter.java:148 registers an Solr Analyzer Attribute to
> the TokenFilter. The Attribute in question is provided by a JAR that
> is embedded in the same bundle. Solr uses "Class.forName(..)" in
> AttributeSource.java:79.
>
> @Reto: it it expected that the SecurityManager denies access to the
> Classloader used by Class.forName(..). Should I move the
> initialization of the Attributes in a doPrivileged(..) block, or is
> something really wrong with the SecurityManager configuration of
> Andreas JVM?
>
> best
> Rupert
>
> On Wed, Oct 16, 2013 at 3:31 PM, Andreas Kuckartz <a....@ping.de>
> wrote:
> > Hi Rupert,
> >
> > thanks again, but unfortunately the integration tests still fail. Log
> > file is appended.
> >
> > Cheers,
> > Andreas
> > ---
> >
> > Rupert Westenthaler:
> >> Hi Andreas
> >>
> >> Based on the new error log the problem was accessing the last
> >> modification date of the FST corpus file outside the doPrivileged()
> >> block.
> >> I have fixed this with http://svn.apache.org/r1532742.
> >>
> >> Would be nice if you could check again
> >>
> >> best
> >> Rupert
> >
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
>
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas, Reto, all
at least we are making progress ...
... loading the FST corpus does now work.
But now the engine is running into an other security related problem
java.security.AccessControlException: access denied
("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
at java.security.AccessController.checkPermission(AccessController.java:559)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1549)
at java.lang.Class.getClassLoader(Class.java:614)
at org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:79)
at org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
at org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
at org.apache.stanbol.enhancer.engines.lucenefstlinking.LinkableTokenFilter.<init>(LinkableTokenFilter.java:148)
at org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.tag(FstLinkingEngine.java:358)
at org.apache.stanbol.enhancer.engines.lucenefstlinking.FstLinkingEngine.computeEnhancements(FstLinkingEngine.java:175)
indicating that the SecurityManager denies Solr access to the Classloader.
LinkableTokenFilter.java:148 registers an Solr Analyzer Attribute to
the TokenFilter. The Attribute in question is provided by a JAR that
is embedded in the same bundle. Solr uses "Class.forName(..)" in
AttributeSource.java:79.
@Reto: it it expected that the SecurityManager denies access to the
Classloader used by Class.forName(..). Should I move the
initialization of the Attributes in a doPrivileged(..) block, or is
something really wrong with the SecurityManager configuration of
Andreas JVM?
best
Rupert
On Wed, Oct 16, 2013 at 3:31 PM, Andreas Kuckartz <a....@ping.de> wrote:
> Hi Rupert,
>
> thanks again, but unfortunately the integration tests still fail. Log
> file is appended.
>
> Cheers,
> Andreas
> ---
>
> Rupert Westenthaler:
>> Hi Andreas
>>
>> Based on the new error log the problem was accessing the last
>> modification date of the FST corpus file outside the doPrivileged()
>> block.
>> I have fixed this with http://svn.apache.org/r1532742.
>>
>> Would be nice if you could check again
>>
>> best
>> Rupert
>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Hi Rupert,
thanks again, but unfortunately the integration tests still fail. Log
file is appended.
Cheers,
Andreas
---
Rupert Westenthaler:
> Hi Andreas
>
> Based on the new error log the problem was accessing the last
> modification date of the FST corpus file outside the doPrivileged()
> block.
> I have fixed this with http://svn.apache.org/r1532742.
>
> Would be nice if you could check again
>
> best
> Rupert
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas
Based on the new error log the problem was accessing the last
modification date of the FST corpus file outside the doPrivileged()
block.
I have fixed this with http://svn.apache.org/r1532742.
Would be nice if you could check again
best
Rupert
On Wed, Oct 16, 2013 at 11:53 AM, Andreas Kuckartz <a....@ping.de> wrote:
> Hi Rupert,
>
> thanks a lot, but unfortunately the integration tests still fail. I will
> send you a new log file in a separate mail.
>
> Some info regarding the environment on Debian unstable:
>
> $ java -version
> java version "1.7.0_25"
> OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4)
> OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> Cheers,
> Andreas
> ---
>
> Rupert Westenthaler:
>> Hi Andreas, all
>>
>> In the meantime Andreas sent me a log file of a failed Stanbol build.
>> Based on the analysis I created STANBOL-1111 [1].
>>
>> With revision 1532653 [2] I committed changes that ensure that all
>> File IO required by the FST linking engine is done within
>> AccessController.doPrivileged(..).
>>
>> @Andreas: can you please verify that this fixes the
>> AccessControlException on your environment. I can not test it as both
>> my build platform as well as the Jenkins builds are not affected by
>> this. If [2] solves this issue for you feel free to mark [1] as
>> resolved.
>>
>> best
>> Rupert
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1177
>> [2] http://svn.apache.org/r1532653
>>
>> On Wed, Oct 9, 2013 at 2:41 PM, Rupert Westenthaler
>> <ru...@gmail.com> wrote:
>>> Hi Andreas,
>>>
>>> looks like a Security related issue of the new FST linking engine. Can
>>> you please find one that exceptions in the log and report is as an
>>> JIRA issue (just assign it to me). [1] provides some additional
>>> information on issues like that
>>>
>>> However AFAIK this could only cause two tests to fail. So it is most
>>> likely not the only cause for your problems.
>>>
>>>
>>> best
>>> Rupert
>>>
>>> [1] http://stanbol.markmail.org/thread/qkkxcecf5ktg7hdu
>>>
>>> On Wed, Oct 9, 2013 at 2:32 PM, Andreas Kuckartz <a....@ping.de> wrote:
>>>> I deleted ~/.m2/repository checked out Stanbol and built it again. Same
>>>> problem.
>>>>
>>>> But I now noticed quite a few of these WARN messages in the output of mvn:
>>>>
>>>> HTTP/1.1 401 Unauthorized [WWW-Authenticate: Basic realm="Apache Stanbol
>>>> authentication needed"
>>>>
>>>> Cheers,
>>>> Andreas
>>>
>>>
>>>
>>> --
>>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>>> | Bodenlehenstraße 11 ++43-699-11108907
>>> | A-5500 Bischofshofen
>>
>>
>>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
Hi Rupert,
thanks a lot, but unfortunately the integration tests still fail. I will
send you a new log file in a separate mail.
Some info regarding the environment on Debian unstable:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Cheers,
Andreas
---
Rupert Westenthaler:
> Hi Andreas, all
>
> In the meantime Andreas sent me a log file of a failed Stanbol build.
> Based on the analysis I created STANBOL-1111 [1].
>
> With revision 1532653 [2] I committed changes that ensure that all
> File IO required by the FST linking engine is done within
> AccessController.doPrivileged(..).
>
> @Andreas: can you please verify that this fixes the
> AccessControlException on your environment. I can not test it as both
> my build platform as well as the Jenkins builds are not affected by
> this. If [2] solves this issue for you feel free to mark [1] as
> resolved.
>
> best
> Rupert
>
> [1] https://issues.apache.org/jira/browse/STANBOL-1177
> [2] http://svn.apache.org/r1532653
>
> On Wed, Oct 9, 2013 at 2:41 PM, Rupert Westenthaler
> <ru...@gmail.com> wrote:
>> Hi Andreas,
>>
>> looks like a Security related issue of the new FST linking engine. Can
>> you please find one that exceptions in the log and report is as an
>> JIRA issue (just assign it to me). [1] provides some additional
>> information on issues like that
>>
>> However AFAIK this could only cause two tests to fail. So it is most
>> likely not the only cause for your problems.
>>
>>
>> best
>> Rupert
>>
>> [1] http://stanbol.markmail.org/thread/qkkxcecf5ktg7hdu
>>
>> On Wed, Oct 9, 2013 at 2:32 PM, Andreas Kuckartz <a....@ping.de> wrote:
>>> I deleted ~/.m2/repository checked out Stanbol and built it again. Same
>>> problem.
>>>
>>> But I now noticed quite a few of these WARN messages in the output of mvn:
>>>
>>> HTTP/1.1 401 Unauthorized [WWW-Authenticate: Basic realm="Apache Stanbol
>>> authentication needed"
>>>
>>> Cheers,
>>> Andreas
>>
>>
>>
>> --
>> | Rupert Westenthaler rupert.westenthaler@gmail.com
>> | Bodenlehenstraße 11 ++43-699-11108907
>> | A-5500 Bischofshofen
>
>
>
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas, all
In the meantime Andreas sent me a log file of a failed Stanbol build.
Based on the analysis I created STANBOL-1111 [1].
With revision 1532653 [2] I committed changes that ensure that all
File IO required by the FST linking engine is done within
AccessController.doPrivileged(..).
@Andreas: can you please verify that this fixes the
AccessControlException on your environment. I can not test it as both
my build platform as well as the Jenkins builds are not affected by
this. If [2] solves this issue for you feel free to mark [1] as
resolved.
best
Rupert
[1] https://issues.apache.org/jira/browse/STANBOL-1177
[2] http://svn.apache.org/r1532653
On Wed, Oct 9, 2013 at 2:41 PM, Rupert Westenthaler
<ru...@gmail.com> wrote:
> Hi Andreas,
>
> looks like a Security related issue of the new FST linking engine. Can
> you please find one that exceptions in the log and report is as an
> JIRA issue (just assign it to me). [1] provides some additional
> information on issues like that
>
> However AFAIK this could only cause two tests to fail. So it is most
> likely not the only cause for your problems.
>
>
> best
> Rupert
>
> [1] http://stanbol.markmail.org/thread/qkkxcecf5ktg7hdu
>
> On Wed, Oct 9, 2013 at 2:32 PM, Andreas Kuckartz <a....@ping.de> wrote:
>> I deleted ~/.m2/repository checked out Stanbol and built it again. Same
>> problem.
>>
>> But I now noticed quite a few of these WARN messages in the output of mvn:
>>
>> HTTP/1.1 401 Unauthorized [WWW-Authenticate: Basic realm="Apache Stanbol
>> authentication needed"
>>
>> Cheers,
>> Andreas
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Andreas,
looks like a Security related issue of the new FST linking engine. Can
you please find one that exceptions in the log and report is as an
JIRA issue (just assign it to me). [1] provides some additional
information on issues like that
However AFAIK this could only cause two tests to fail. So it is most
likely not the only cause for your problems.
best
Rupert
[1] http://stanbol.markmail.org/thread/qkkxcecf5ktg7hdu
On Wed, Oct 9, 2013 at 2:32 PM, Andreas Kuckartz <a....@ping.de> wrote:
> I deleted ~/.m2/repository checked out Stanbol and built it again. Same
> problem.
>
> But I now noticed quite a few of these WARN messages in the output of mvn:
>
> HTTP/1.1 401 Unauthorized [WWW-Authenticate: Basic realm="Apache Stanbol
> authentication needed"
>
> Cheers,
> Andreas
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Update on integration test problems
Posted by Andreas Kuckartz <a....@ping.de>.
I deleted ~/.m2/repository checked out Stanbol and built it again. Same
problem.
But I now noticed quite a few of these WARN messages in the output of mvn:
HTTP/1.1 401 Unauthorized [WWW-Authenticate: Basic realm="Apache Stanbol
authentication needed"
Cheers,
Andreas