You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by Andy Gross <an...@andygross.org> on 2009/03/16 18:25:44 UTC
Trouble running examples
Droids seems like a very interesting project upon which I'd like to
base
some of my custom web crawlers - but I'm running into this error
when I try
to run the examples. The file droids-core/src/test/java/
org/apache/droids/examples/SimpleRuntime.java, for example, throws
this
exception. I've spent awhile trying to figure out what the
problem is, to
no avail.
I've verified with TCPDump that running the example never initiates
any HTTP activity. I'm running droids-trunk, svn revision 754973, on
Mac
OSX 10.5.6.
Can anyone point me in the right direction?
2009-03-16
14:24:04,107 INFO [main] - <Number of concurrent workers: 5>
2009-03-16
14:24:04,113 INFO [main] - <Finshed invocation, waiting for
workers to
finish.>
2009-03-16 14:24:04,113 INFO [pool-1-thread-2] - <SHUTTING DOWN>
2009-03-16 14:24:04,114 INFO [pool-1-thread-2] - <All threads have
finished.
(elapsed: 8)>
2009-03-16 14:24:04,114 INFO [pool-1-thread-2] - <FINISHED!!!>
2009-03-16 14:24:04,760 WARN [pool-1-thread-1] - <Connection pool shut
down.>
java.lang.IllegalStateException: Connection pool shut down.
at
org
.apache
.http
.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:
284)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
$1.getPoolEntry(ConnPoolByRoute.java:238)
at
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
$1.getConnection(ThreadSafeClientConnManager.java:175)
at
org
.apache
.http
.impl
.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
at
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
555)
at
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
487)
at
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
465)
at
org
.apache
.droids
.protocol
.http.HttpClientContentLoader.exists(HttpClientContentLoader.java:58)
at org.apache.droids.norobots.NoRobotClient.parse(NoRobotClient.java:
98)
at
org
.apache.droids.protocol.http.HttpProtocol.isAllowed(HttpProtocol.java:
102)
at
org
.apache
.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:53)
at
org
.apache
.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:36)
at org.apache.droids.impl.MultiThreadedTaskMaster
$WorkerRunner.run(MultiThreadedTaskMaster.java:280)
at
java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:637)
2009-03-16 14:24:04,764 INFO
[pool-1-thread-1] - <SHUTTING DOWN>
2009-03-16 14:24:04,764 INFO
[pool-1-thread-1] - <All threads have
finished. (elapsed: 658)>
2009-03-16
14:24:04,764 INFO [pool-1-thread-1] - <FINISHED!!!>
Re: Trouble running examples
Posted by Thorsten Scherler <sc...@gmail.com>.
On Mon, 2009-03-23 at 20:52 +0100, Oleg Kalnichevski wrote:
> Thorsten Scherler wrote:
> > On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
> >> Oleg Kalnichevski wrote:
> > ...
> >> The problem is not caused by the HTTP transport. The
> >> MultiThreadedTaskMaster appears to have synchronization issues and
> >> should also be reviewed for thread-safety.
> >
> > Ok, do you have any method in special as possible candidates? Will now
> > look into it and hopefully find the problem.
> >
> > salu2
>
> I have looked at MultiThreadedTaskMaster quite extensively and got an
> impression that it simply did not attempt to synchronize access to its
> instance variables at all despite the fact they could be accessed from
> multiple threads of execution. I'll try to find time to propose a fix
> for the issue, but cannot really tell when I can manage to get around to
> doing that.
>
Thank you very much Oleg, looking forward.
I ATM preparing some small example for ApacheCon and your recent change
shows the sequential capacity so that is good and brings back a running
example.
salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>
--
Thorsten Scherler <scherler.at.gmail.com>
Open Source <consulting, training and solutions>
Re: Trouble running examples
Posted by Thorsten Scherler <sc...@gmail.com>.
On Mon, 2009-03-23 at 20:52 +0100, Oleg Kalnichevski wrote:
> Thorsten Scherler wrote:
> > On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
> >> Oleg Kalnichevski wrote:
> > ...
> >> The problem is not caused by the HTTP transport. The
> >> MultiThreadedTaskMaster appears to have synchronization issues and
> >> should also be reviewed for thread-safety.
> >
> > Ok, do you have any method in special as possible candidates? Will now
> > look into it and hopefully find the problem.
> >
> > salu2
>
> I have looked at MultiThreadedTaskMaster quite extensively and got an
> impression that it simply did not attempt to synchronize access to its
> instance variables at all despite the fact they could be accessed from
> multiple threads of execution. I'll try to find time to propose a fix
> for the issue, but cannot really tell when I can manage to get around to
> doing that.
>
Thank you very much Oleg, looking forward.
I ATM preparing some small example for ApacheCon and your recent change
shows the sequential capacity so that is good and brings back a running
example.
salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>
--
Thorsten Scherler <scherler.at.gmail.com>
Open Source <consulting, training and solutions>
Re: Trouble running examples
Posted by Oleg Kalnichevski <ol...@apache.org>.
Thorsten Scherler wrote:
> On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
>> Oleg Kalnichevski wrote:
> ...
>> The problem is not caused by the HTTP transport. The
>> MultiThreadedTaskMaster appears to have synchronization issues and
>> should also be reviewed for thread-safety.
>
> Ok, do you have any method in special as possible candidates? Will now
> look into it and hopefully find the problem.
>
> salu2
I have looked at MultiThreadedTaskMaster quite extensively and got an
impression that it simply did not attempt to synchronize access to its
instance variables at all despite the fact they could be accessed from
multiple threads of execution. I'll try to find time to propose a fix
for the issue, but cannot really tell when I can manage to get around to
doing that.
Oleg
Re: Trouble running examples
Posted by Thorsten Scherler <th...@apache.org>.
On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
> Oleg Kalnichevski wrote:
...
> >
>
> The problem is not caused by the HTTP transport. The
> MultiThreadedTaskMaster appears to have synchronization issues and
> should also be reviewed for thread-safety.
Ok, do you have any method in special as possible candidates? Will now
look into it and hopefully find the problem.
salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>
Re: Trouble running examples
Posted by Oleg Kalnichevski <ol...@apache.org>.
Oleg Kalnichevski wrote:
> Thorsten Scherler wrote:
>> On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
>>> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
>>>> Droids seems like a very interesting project upon which I'd like to
>>>> base some of my custom web crawlers - but I'm running into this
>>>> error when I try to run the examples. The file
>>>> droids-core/src/test/java/
>>>> org/apache/droids/examples/SimpleRuntime.java, for example, throws
>>>> this exception. I've spent awhile trying to figure out what the
>>>> problem is, to no avail.
>>>>
>>>> I've verified with TCPDump that running the example never initiates
>>>> any HTTP activity. I'm running droids-trunk, svn revision 754973,
>>>> on Mac OSX 10.5.6.
>>> I just tried and I have a similar problem. Not sure ATM why but I can
>>> reproduce the problem.
>>>
>>> I do not get an exception like you but rather: <Finshed invocation,
>>> waiting for workers to finish.>
>>> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
>>> (org.apache.http.NoHttpResponseException) caught when processing
>>> request: The target server failed to respond>
>>
>> If I do Index:
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> ===================================================================
>> ---
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> (revision 755424)
>> +++
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> (working copy)
>> @@ -98,6 +98,7 @@
>> DroidsHttpClient httpclient = new DroidsHttpClient(params);
>> HttpProtocol httpProtocol = new HttpProtocol(httpclient);
>> + httpProtocol.setForceAllow(true);
>> protocolFactory.setMap(new HashMap<String, Object>());
>>
>> I as well get. Seems that the http pool is not working.
>> Oleg do you have an idea?
>>
>
> Thorsten,
>
> I'll look into it.
>
> Oleg
>
The problem is not caused by the HTTP transport. The
MultiThreadedTaskMaster appears to have synchronization issues and
should also be reviewed for thread-safety.
Oleg
>
>> java.lang.IllegalStateException: Connection pool shut down.
>> at
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
>>
>> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
>> $1.getPoolEntry(ConnPoolByRoute.java:238)
>> at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
>> $1.getConnection(ThreadSafeClientConnManager.java:175)
>> at
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
>>
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
>>
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
>>
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
>>
>> at
>> org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
>> at
>> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
>>
>> at
>> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
>>
>> at org.apache.droids.impl.MultiThreadedTaskMaster
>> $WorkerRunner.run(MultiThreadedTaskMaster.java:280)
>> at java.util.concurrent.ThreadPoolExecutor
>> $Worker.runTask(ThreadPoolExecutor.java:885)
>> at java.util.concurrent.ThreadPoolExecutor
>> $Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>> salu2
>
Re: Trouble running examples
Posted by Oleg Kalnichevski <ol...@apache.org>.
Thorsten Scherler wrote:
> On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
>> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
>>> Droids seems like a very interesting project upon which I'd like to
>>> base some of my custom web crawlers - but I'm running into this error
>>> when I try to run the examples. The file droids-core/src/test/java/
>>> org/apache/droids/examples/SimpleRuntime.java, for example, throws
>>> this exception. I've spent awhile trying to figure out what the
>>> problem is, to no avail.
>>>
>>> I've verified with TCPDump that running the example never initiates
>>> any HTTP activity. I'm running droids-trunk, svn revision 754973, on
>>> Mac OSX 10.5.6.
>> I just tried and I have a similar problem. Not sure ATM why but I can
>> reproduce the problem.
>>
>> I do not get an exception like you but rather:
>> <Finshed invocation, waiting for workers to finish.>
>> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
>> (org.apache.http.NoHttpResponseException) caught when processing
>> request: The target server failed to respond>
>
> If I do
> Index:
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> ===================================================================
> ---
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> (revision 755424)
> +++
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> (working copy)
> @@ -98,6 +98,7 @@
> DroidsHttpClient httpclient = new DroidsHttpClient(params);
>
> HttpProtocol httpProtocol = new HttpProtocol(httpclient);
> + httpProtocol.setForceAllow(true);
> protocolFactory.setMap(new HashMap<String, Object>());
>
> I as well get. Seems that the http pool is not working.
>
> Oleg do you have an idea?
>
Thorsten,
I'll look into it.
Oleg
> java.lang.IllegalStateException: Connection pool shut down.
> at
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
> $1.getPoolEntry(ConnPoolByRoute.java:238)
> at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> $1.getConnection(ThreadSafeClientConnManager.java:175)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
> at
> org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
> at
> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
> at
> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
> at org.apache.droids.impl.MultiThreadedTaskMaster
> $WorkerRunner.run(MultiThreadedTaskMaster.java:280)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
>> salu2
Re: Trouble running examples
Posted by Thorsten Scherler <th...@apache.org>.
On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
> > Droids seems like a very interesting project upon which I'd like to
> > base some of my custom web crawlers - but I'm running into this error
> > when I try to run the examples. The file droids-core/src/test/java/
> > org/apache/droids/examples/SimpleRuntime.java, for example, throws
> > this exception. I've spent awhile trying to figure out what the
> > problem is, to no avail.
> >
> > I've verified with TCPDump that running the example never initiates
> > any HTTP activity. I'm running droids-trunk, svn revision 754973, on
> > Mac OSX 10.5.6.
>
> I just tried and I have a similar problem. Not sure ATM why but I can
> reproduce the problem.
>
> I do not get an exception like you but rather:
> <Finshed invocation, waiting for workers to finish.>
> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
> (org.apache.http.NoHttpResponseException) caught when processing
> request: The target server failed to respond>
If I do
Index:
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
===================================================================
---
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
(revision 755424)
+++
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
(working copy)
@@ -98,6 +98,7 @@
DroidsHttpClient httpclient = new DroidsHttpClient(params);
HttpProtocol httpProtocol = new HttpProtocol(httpclient);
+ httpProtocol.setForceAllow(true);
protocolFactory.setMap(new HashMap<String, Object>());
I as well get. Seems that the http pool is not working.
Oleg do you have an idea?
java.lang.IllegalStateException: Connection pool shut down.
at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
$1.getPoolEntry(ConnPoolByRoute.java:238)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
$1.getConnection(ThreadSafeClientConnManager.java:175)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
at
org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
at
org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
at
org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
at org.apache.droids.impl.MultiThreadedTaskMaster
$WorkerRunner.run(MultiThreadedTaskMaster.java:280)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
> salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>
Re: Trouble running examples
Posted by Thorsten Scherler <th...@apache.org>.
On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
> Droids seems like a very interesting project upon which I'd like to
> base some of my custom web crawlers - but I'm running into this error
> when I try to run the examples. The file droids-core/src/test/java/
> org/apache/droids/examples/SimpleRuntime.java, for example, throws
> this exception. I've spent awhile trying to figure out what the
> problem is, to no avail.
>
> I've verified with TCPDump that running the example never initiates
> any HTTP activity. I'm running droids-trunk, svn revision 754973, on
> Mac OSX 10.5.6.
I just tried and I have a similar problem. Not sure ATM why but I can
reproduce the problem.
I do not get an exception like you but rather:
<Finshed invocation, waiting for workers to finish.>
2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
(org.apache.http.NoHttpResponseException) caught when processing
request: The target server failed to respond>
salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>