You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by "Mingfai Ma (JIRA)" <ji...@apache.org> on 2009/04/18 17:47:14 UTC
[jira] Created: (DROIDS-47) SimpleTaskQueue is very slow to init
when there is a large number of seed URLs
SimpleTaskQueue is very slow to init when there is a large number of seed URLs
------------------------------------------------------------------------------
Key: DROIDS-47
URL: https://issues.apache.org/jira/browse/DROIDS-47
Project: Droids
Issue Type: Improvement
Components: core
Affects Versions: 0.01
Reporter: Mingfai Ma
when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DROIDS-47) SimpleTaskQueue is very slow to init
when there is a large number of seed URLs
Posted by "Thorsten Scherler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thorsten Scherler reassigned DROIDS-47:
---------------------------------------
Assignee: Thorsten Scherler
> SimpleTaskQueue is very slow to init when there is a large number of seed URLs
> ------------------------------------------------------------------------------
>
> Key: DROIDS-47
> URL: https://issues.apache.org/jira/browse/DROIDS-47
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Assignee: Thorsten Scherler
> Attachments: DROIDS-47.patch, DROIDS-47_profile-after.png, DROIDS-47_profile.png
>
>
> when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DROIDS-47) SimpleTaskQueue is very slow to init
when there is a large number of seed URLs
Posted by "Thorsten Scherler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thorsten Scherler closed DROIDS-47.
-----------------------------------
Resolution: Fixed
Assignee: (was: Thorsten Scherler)
Committed revision 772801.
Thanks Mingfai Ma keep up the good work.
> SimpleTaskQueue is very slow to init when there is a large number of seed URLs
> ------------------------------------------------------------------------------
>
> Key: DROIDS-47
> URL: https://issues.apache.org/jira/browse/DROIDS-47
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Attachments: DROIDS-47.patch, DROIDS-47_profile-after.png, DROIDS-47_profile.png
>
>
> when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DROIDS-47) SimpleTaskQueue is very slow to init
when there is a large number of seed URLs
Posted by "Mingfai Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingfai Ma updated DROIDS-47:
-----------------------------
Attachment: DROIDS-47_profile.png
when there are more than 50,000 items, the ConcurrentLinkedQueue is very slow.
> SimpleTaskQueue is very slow to init when there is a large number of seed URLs
> ------------------------------------------------------------------------------
>
> Key: DROIDS-47
> URL: https://issues.apache.org/jira/browse/DROIDS-47
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Attachments: DROIDS-47_profile.png
>
>
> when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DROIDS-47) SimpleTaskQueue is very slow to init
when there is a large number of seed URLs
Posted by "Mingfai Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DROIDS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingfai Ma updated DROIDS-47:
-----------------------------
Attachment: DROIDS-47.patch
DROIDS-47_profile-after.png
be honest, i think it does not make sense to do a size checking that throw an exception.
when a crawling job is being run, links are keep adding to the queue. it make sense to have a limit to throttle memory usage, so one may use a bounding queue and additional links cannot be added before links are processed. However, throwing exception means the extractor thread will get exception, the links will be discarded, and either the extractor thread fails or the whole Droids job fails. (hope it would be the former case)
my patch does the following:
- commented the queue size checking logic
- added a constructor to allow user to supply a Queue. if anyone need a queue that throw exception when it is full, he may supply one.
- commented the testMaxSize method in TestSimpleQueue, you may want to delete the content instead
> SimpleTaskQueue is very slow to init when there is a large number of seed URLs
> ------------------------------------------------------------------------------
>
> Key: DROIDS-47
> URL: https://issues.apache.org/jira/browse/DROIDS-47
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Attachments: DROIDS-47.patch, DROIDS-47_profile-after.png, DROIDS-47_profile.png
>
>
> when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.