You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by "Eugen Paraschiv (JIRA)" <ji...@apache.org> on 2011/02/15 21:32:57 UTC
[jira] Created: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Better exception handling in SimpleTaskQueue - merge
----------------------------------------------------
Key: DROIDS-120
URL: https://issues.apache.org/jira/browse/DROIDS-120
Project: Droids
Issue Type: Improvement
Components: core
Affects Versions: 0.0.1
Reporter: Eugen Paraschiv
Fix For: 0.0.1
In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugen Paraschiv updated DROIDS-120:
-----------------------------------
Attachment: DROIDS-120_v1.patch
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v1.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugen Paraschiv resolved DROIDS-120.
------------------------------------
Resolution: Fixed
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.2
> Reporter: Eugen Paraschiv
> Fix For: 0.0.2
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Thorsten Scherler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997765#comment-12997765 ]
Thorsten Scherler commented on DROIDS-120:
------------------------------------------
I totally agree that we should implement better exception handling in the merge method. You are right Eugen, that it cannot be that the merge will stop droids. For now I would say log the exception and go on and in the near future Bertil may present the finer ExceptionHandler. WDYT?
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997490#comment-12997490 ]
Bertil Chapuis commented on DROIDS-120:
---------------------------------------
I agree. Loosing all the Links is not acceptable. However I thinks it's important to be consistent with the way exceptions are handled. In my opinion it could make sense to attach the TaskExceptionHandler with the Droid instead of the TaskMaster. This will introduce some minor changes to the api but it will allows to handle each exceptions. I may provide a patch, let me know if you think it's a good idea.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995016#comment-12995016 ]
Bertil Chapuis commented on DROIDS-120:
---------------------------------------
There is a plan to replace TaskQueue by java.util.Queue instead (DROIDS-56). You may want to have a look at the patch and see if some modification have to be done to handle the exceptions more properly.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995034#comment-12995034 ]
Eugen Paraschiv commented on DROIDS-120:
----------------------------------------
Thank you for the quick feedback, and for making me aware of the direction the Droids queue implementation is evolving towards. Here's my 2 cents on the issue - DROIDS-56 is a major restructuring of the codebase, and it already has a lifespan of several months. It is clearly a good direction but it also may or may not make it into 0.0.1.
This issue is a minor but beneficial change on the current codebase, dealing with a single method in a single class but improving the overall experience. So my view is that the two issues do not exclude each other - if the patch for this issue is OK, then it will improve the codebase now, and when DROIDS-56 is going to be committed, then this merge collection will simply be addAll on the queue which will also address the problem.
Hope this makes sense.
Thanks again.
Eugen.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v1.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995272#comment-12995272 ]
Bertil Chapuis commented on DROIDS-120:
---------------------------------------
I may be wrong but I think the exception should be handled by the TaskExceptionHandler as the other exceptions which could occur in droids.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v1.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugen Paraschiv updated DROIDS-120:
-----------------------------------
Attachment: DROIDS-120_v2.patch
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996027#comment-12996027 ]
Eugen Paraschiv commented on DROIDS-120:
----------------------------------------
The way I see it, the merge(Collection) method is called in CrawlingWorker:
droid.getQueue().merge( outlinks );
Now, if an exception is thrown when merging one of these Links, and if the exception is allowed to bubble up the stack, then the remaining links in that collection will no longer be processed. I think it's perfectly OK to lose the Link that had the problem and generated the exception, but not OK to loose the entire collection because of a bad link. That is why I proposed to handle the exception for each Link individually - because of this distinct possibility of loosing the entire collection (which does happen in production).
Thanks for the feedback.
Eugen.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v1.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998035#comment-12998035 ]
Bertil Chapuis commented on DROIDS-120:
---------------------------------------
Ok for me. Since the SimpleTaskQueue will probably be replaced by java.util.Queue working on a finer ExceptionHandler is not the first priority. I applied the patch and rescheduled the ticket for 0.0.2.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bertil Chapuis updated DROIDS-120:
----------------------------------
Fix Version/s: (was: 0.0.1)
0.0.2
Affects Version/s: (was: 0.0.1)
0.0.2
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.2
> Reporter: Eugen Paraschiv
> Fix For: 0.0.2
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998034#comment-12998034 ]
Bertil Chapuis commented on DROIDS-120:
---------------------------------------
Ok for me. Since the SimpleTaskQueue will probably be replaced by java.util.Queue working on a finer ExceptionHandler is not the first priority. I applied the patch and rescheduled the ticket for 0.0.2.
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Eugen Paraschiv (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugen Paraschiv updated DROIDS-120:
-----------------------------------
Attachment: (was: DROIDS-120_v1.patch)
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DROIDS-120) Better exception handling in
SimpleTaskQueue - merge
Posted by "Bertil Chapuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bertil Chapuis updated DROIDS-120:
----------------------------------
Fix Version/s: (was: 0.0.2)
0.0.1
Affects Version/s: (was: 0.0.2)
0.0.1
> Better exception handling in SimpleTaskQueue - merge
> ----------------------------------------------------
>
> Key: DROIDS-120
> URL: https://issues.apache.org/jira/browse/DROIDS-120
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.0.1
> Reporter: Eugen Paraschiv
> Fix For: 0.0.1
>
> Attachments: DROIDS-120_v2.patch
>
>
> In SimpleTaskQueue - merge, any exception in any of the individual tasks would end up in closing down the entire process and loosing all the results in that particular collection.
> This should not be the case - when iterating over a (possibly large) collection, where each result is relevant, an exception should be logged and the iteration should continue on to the next element.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira