You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by Javier Puerto <jp...@gmail.com> on 2009/11/06 13:29:46 UTC
HandlerFactory fails with multithreaded implementation
Hi, I'm working with Droids and made some URL crawlers to save a lot of web
pages in disk. In JUnit test, I run a little http server and crawl 20 pages,
the most times everything works ok but in rare cases I get an error. I found
the problem in the HandlerFactory implementation, in the example the call to
handlers is like this:
protected void handle(ContentEntity entity, Link link)
throws DroidsException, IOException
{
droid.getHandlerFactory().handle(link.getURI(), entity);
}
If two or more workers is trying to handle at same time, the HandlerFactory
will handle the all with the same instance of the handler. The solution
could be saving memory or improving performance.
The first solution could be implemented adding a "synchronized" to
HandlerFactory.handle like this.
public synchronized boolean handle(URI uri, ContentEntity entity)
throws DroidsException, IOException {
for (Handler handler : getMap().values()) {
handler.handle(uri, entity);
}
return true;
}
Only one handler to share with all workers but this solution is a
performance killer. The other approx should be the opposite, each worker had
his own instance of
the handlerfactory or handler.
Solution that you think might be more appropriate?
Salu2.
Re: HandlerFactory fails with multithreaded implementation
Posted by Javier Puerto <jp...@gmail.com>.
I'm working on a solution for this problem and I think that the best fix
would be to move the HandlerFactory object from CrawlingDroid to
CrawlingWorker. With the cloneable solution, every handler must override the
clone() method and implements the interface Cloneable.
I think that moving the factory to the worker could be the best solution,
because with each thread (worker) you will have a new factory object with
new handlers and you we can to forget about the clone implementation. Also,
the clone solution will create and destroy all handlers for each target
handle in each thread. Opposite, the move fix will take more memory but all
the threads will have it's own handlers.
I have a problem about this fix, I wonder where can be the HandlerFactory
init for the Worker.
What do you think about?
Salu2.
Re: HandlerFactory fails with multithreaded implementation
Posted by Javier Puerto <jp...@gmail.com>.
2009/11/12 Chapuis Bertil <bc...@agimem.com>
> Not in a short scope, I am quite busy this month. But Javier could create a
> ticket for the issue.
>
> Best regards,
>
> Bertil Chapuis
>
Ticket created on https://issues.apache.org/jira/browse/DROIDS-68
I add the synchroniced fix as a workaround in the ticket for a fast
solution. The cloneable fix is better but I'm busy too.
Best regards.
Re: HandlerFactory fails with multithreaded implementation
Posted by Chapuis Bertil <bc...@agimem.com>.
On Wed, Nov 11, 2009 at 9:25 AM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:
> On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote:
> > I had the same problem and solved it in my handler's implementation by
> only
> > using local variables and limited concurrent accesses. As I understand
> the
> > issue the same limitation could occurs with a custom CrawlingDroid
> > implementation since all workers are using the same Droid. A nice fix
> could
> > be to make the Droid and the GenericFactory abstractions clonable and
> > invoking the clone method in the Worker's constructor.
>
> Could you provide a patch?
Not in a short scope, I am quite busy this month. But Javier could create a
ticket for the issue.
Best regards,
Bertil Chapuis
Re: HandlerFactory fails with multithreaded implementation
Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote:
> I had the same problem and solved it in my handler's implementation by only
> using local variables and limited concurrent accesses. As I understand the
> issue the same limitation could occurs with a custom CrawlingDroid
> implementation since all workers are using the same Droid. A nice fix could
> be to make the Droid and the GenericFactory abstractions clonable and
> invoking the clone method in the Worker's constructor.
Could you provide a patch? When I talked with Javier we discussed this
option as well and IMO it is an elegant solution.
Thanks for your feedback Bertil.
salu2
>
> Best regards,
>
> Bertil Chapuis
>
>
> On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
> thorsten.scherler.ext@juntadeandalucia.es> wrote:
>
> > On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> > web
> > > pages in disk. In JUnit test, I run a little http server and crawl 20
> > pages,
> > > the most times everything works ok but in rare cases I get an error. I
> > found
> > > the problem in the HandlerFactory implementation, in the example the call
> > to
> > > handlers is like this:
> > >
> > > protected void handle(ContentEntity entity, Link link)
> > > throws DroidsException, IOException
> > > {
> > > droid.getHandlerFactory().handle(link.getURI(), entity);
> > > }
> > >
> > >
> > > If two or more workers is trying to handle at same time, the
> > HandlerFactory
> > > will handle the all with the same instance of the handler. The solution
> > > could be saving memory or improving performance.
> > >
> > > The first solution could be implemented adding a "synchronized" to
> > > HandlerFactory.handle like this.
> > >
> > > public synchronized boolean handle(URI uri, ContentEntity entity)
> > > throws DroidsException, IOException {
> > > for (Handler handler : getMap().values()) {
> > > handler.handle(uri, entity);
> > > }
> > > return true;
> > > }
> > >
> > > Only one handler to share with all workers but this solution is a
> > > performance killer. The other approx should be the opposite, each worker
> > had
> > > his own instance of
> > > the handlerfactory or handler.
> > >
> > > Solution that you think might be more appropriate?
> >
> > It depends on the usecase I guess. However I think the second option is
> > the more common solution.
> >
> > salu2
> >
> > >
> > > Salu2.
> > --
> > Thorsten Scherler <thorsten.at.apache.org>
> > Open Source Java <consulting, training and solutions>
> >
> > Sociedad Andaluza para el Desarrollo de la Sociedad
> > de la Información, S.A.U. (SADESI)
> >
> >
> >
> >
> >
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>
Sociedad Andaluza para el Desarrollo de la Sociedad
de la Información, S.A.U. (SADESI)
Re: HandlerFactory fails with multithreaded implementation
Posted by Chapuis Bertil <bc...@agimem.com>.
I had the same problem and solved it in my handler's implementation by only
using local variables and limited concurrent accesses. As I understand the
issue the same limitation could occurs with a custom CrawlingDroid
implementation since all workers are using the same Droid. A nice fix could
be to make the Droid and the GenericFactory abstractions clonable and
invoking the clone method in the Worker's constructor.
Best regards,
Bertil Chapuis
On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:
> On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> web
> > pages in disk. In JUnit test, I run a little http server and crawl 20
> pages,
> > the most times everything works ok but in rare cases I get an error. I
> found
> > the problem in the HandlerFactory implementation, in the example the call
> to
> > handlers is like this:
> >
> > protected void handle(ContentEntity entity, Link link)
> > throws DroidsException, IOException
> > {
> > droid.getHandlerFactory().handle(link.getURI(), entity);
> > }
> >
> >
> > If two or more workers is trying to handle at same time, the
> HandlerFactory
> > will handle the all with the same instance of the handler. The solution
> > could be saving memory or improving performance.
> >
> > The first solution could be implemented adding a "synchronized" to
> > HandlerFactory.handle like this.
> >
> > public synchronized boolean handle(URI uri, ContentEntity entity)
> > throws DroidsException, IOException {
> > for (Handler handler : getMap().values()) {
> > handler.handle(uri, entity);
> > }
> > return true;
> > }
> >
> > Only one handler to share with all workers but this solution is a
> > performance killer. The other approx should be the opposite, each worker
> had
> > his own instance of
> > the handlerfactory or handler.
> >
> > Solution that you think might be more appropriate?
>
> It depends on the usecase I guess. However I think the second option is
> the more common solution.
>
> salu2
>
> >
> > Salu2.
> --
> Thorsten Scherler <thorsten.at.apache.org>
> Open Source Java <consulting, training and solutions>
>
> Sociedad Andaluza para el Desarrollo de la Sociedad
> de la Información, S.A.U. (SADESI)
>
>
>
>
>
Re: HandlerFactory fails with multithreaded implementation
Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> Hi, I'm working with Droids and made some URL crawlers to save a lot of web
> pages in disk. In JUnit test, I run a little http server and crawl 20 pages,
> the most times everything works ok but in rare cases I get an error. I found
> the problem in the HandlerFactory implementation, in the example the call to
> handlers is like this:
>
> protected void handle(ContentEntity entity, Link link)
> throws DroidsException, IOException
> {
> droid.getHandlerFactory().handle(link.getURI(), entity);
> }
>
>
> If two or more workers is trying to handle at same time, the HandlerFactory
> will handle the all with the same instance of the handler. The solution
> could be saving memory or improving performance.
>
> The first solution could be implemented adding a "synchronized" to
> HandlerFactory.handle like this.
>
> public synchronized boolean handle(URI uri, ContentEntity entity)
> throws DroidsException, IOException {
> for (Handler handler : getMap().values()) {
> handler.handle(uri, entity);
> }
> return true;
> }
>
> Only one handler to share with all workers but this solution is a
> performance killer. The other approx should be the opposite, each worker had
> his own instance of
> the handlerfactory or handler.
>
> Solution that you think might be more appropriate?
It depends on the usecase I guess. However I think the second option is
the more common solution.
salu2
>
> Salu2.
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>
Sociedad Andaluza para el Desarrollo de la Sociedad
de la Información, S.A.U. (SADESI)