You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@any23.apache.org by "Timothy Potter (Created) (JIRA)" <ji...@apache.org> on 2012/04/12 18:35:18 UTC
[jira] [Created] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Improve runtime of the Microdata extractor on documents with many relations.
----------------------------------------------------------------------------
Key: ANY23-75
URL: https://issues.apache.org/jira/browse/ANY23-75
Project: Apache Any23
Issue Type: Improvement
Reporter: Timothy Potter
Priority: Minor
I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Lewis John McGibbney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney updated ANY23-75:
--------------------------------------
Affects Version/s: 0.7.0
Fix Version/s: 0.7.0
Setting for 0.7.0-incubating.
Is it possible for you to explain a bit about the patch and the underlying reason as to why the existing parser implementation seems to clog up? Also this is really trivial but can you please have a look at the coding format if it differs or not. My initial thoughts of the patch are great, it's a nice one to have caught, but some additional explanation would really help us out. Thank you very much. Lewis
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Timothy Potter (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254621#comment-13254621 ]
Timothy Potter commented on ANY23-75:
-------------------------------------
Thanks Michele.
I'll explain the change anyway:
The current implementation is slow on pages with lots of items because it does nested iterations over all itemscope and itemprops under the given scopeNode. In the inner loop it builds XPaths strings for each node to test if the nodes are related. This test condition is inherent in the tree structure of the DOM. The patch changes the code to do a traversal of the DOM tree limited to nodes only in the given scopeNode. Reading the code itself is probably the best way to understanding the change.
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Timothy Potter (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Potter updated ANY23-75:
--------------------------------
Attachment: MicrodataParser.diff
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Reporter: Timothy Potter
> Priority: Minor
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287867#comment-13287867 ]
Hudson commented on ANY23-75:
-----------------------------
Integrated in Any23-trunk #220 (See [https://builds.apache.org/job/Any23-trunk/220/])
Improved MicrodataParser performances. Related to issue #ANY23-75. (Revision 1345154)
Result = SUCCESS
mostarda :
Files :
* /incubator/any23/trunk/core/src/main/java/org/apache/any23/extractor/microdata/MicrodataParser.java
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Michele Mostarda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michele Mostarda reassigned ANY23-75:
-------------------------------------
Assignee: Michele Mostarda
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Michele Mostarda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michele Mostarda resolved ANY23-75.
-----------------------------------
Resolution: Fixed
Fixed @1345154
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney closed ANY23-75.
-------------------------------------
Bulk close for 0.7.0-incubating release
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Michele Mostarda (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michele Mostarda updated ANY23-75:
----------------------------------
Priority: Major (was: Minor)
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-75) Improve runtime of the Microdata
extractor on documents with many relations.
Posted by "Michele Mostarda (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254570#comment-13254570 ]
Michele Mostarda commented on ANY23-75:
---------------------------------------
I read the patch, it sounds really clear, it is an optimization to improve performances while processing detecting nested node paths. I will integrate it in a couple of days.
> Improve runtime of the Microdata extractor on documents with many relations.
> ----------------------------------------------------------------------------
>
> Key: ANY23-75
> URL: https://issues.apache.org/jira/browse/ANY23-75
> Project: Apache Any23
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Timothy Potter
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: MicrodataParser.diff
>
>
> I've been running Any23 on a big web crawler dump. I found for certain documents with a lot of Microdata relations the method MicrodataParser.getItemProps() becomes very slow. As a result, processing one document can take several minutes. An example of a problematic page can be seen here: http://dreamtime.fftunes.com/
> I'll attach a patch for the method that greatly improves the performance of this method. I was wondering if someone could have a look at it and include it in the next release if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira