You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@any23.apache.org by "Michele Mostarda (Created) (JIRA)" <ji...@apache.org> on 2012/04/20 15:10:39 UTC
[jira] [Created] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Facing a infinite loop problem in version 0.6.1 - Verify
--------------------------------------------------------
Key: ANY23-77
URL: https://issues.apache.org/jira/browse/ANY23-77
Project: Apache Any23
Issue Type: Bug
Reporter: Michele Mostarda
Assignee: Michele Mostarda
The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
Java Code:
Client c = Client.create();
System.out.println("Downloading file.");
InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
FileOutputStream out = null;
File f = new File("urlResource");
try {
out = new FileOutputStream(f);
IOUtils.copy(in, out);
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(out);
}
System.out.println("File downloaded.");
System.out.println("Starting extraction.");
FileDocumentSource doc = new FileDocumentSource(f);
TurtleWriter tw = new TurtleWriter(System.out);
Any23 ext = new Any23();
try {
ext.extract(doc, tw);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney closed ANY23-77.
-------------------------------------
Bulk close for 0.7.0-incubating release
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Components: service
> Affects Versions: 0.7.0
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Ondrej Klimpera (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258328#comment-13258328 ]
Ondrej Klimpera commented on ANY23-77:
--------------------------------------
Hello,
digged a little bit deeper in this problem. This loop is not infinite it will end, but it takes too long to process. On my notebook it took about 2 hours to process this resource (about 2 MB). The problem lies in very ineffective implementation of microdata parsing. What I found so far, the class org.deri.any23.extractor.html.DomUtils with method getXPathForNode() is using recursion that is called really often and this causes so long processing time.
As a raw fix I created a local cahe for this method to store mostly used xPath statements and cut the processing time to 13 minutes. It's not ideal, but at least it shows the way of making microdata parsing much more effective - by adding smth like local caching to extractor (not only to DomUtils).
Please let me know if there is any desire to fix this.
Thanks for dealing with it :)
Regards
Ondrej Klimpera
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Szymon Danielczyk (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258337#comment-13258337 ]
Szymon Danielczyk commented on ANY23-77:
----------------------------------------
Hi Ondrej
Very good catch ,analysis and possible solution/improvements
I would say there is an interest in this as the issue already surfaced few times
I remember at least one or two people already hit this problem before you
Big Thank you for investigating this.
All the Best
Szymon
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Michele Mostarda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michele Mostarda resolved ANY23-77.
-----------------------------------
Resolution: Fixed
Reopen the issue if the performance improvement cannot be verified.
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Michele Mostarda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259065#comment-13259065 ]
Michele Mostarda commented on ANY23-77:
---------------------------------------
I've the improved nesting detection code removing the logic relying on XPath strings, then I tested Any23 on the reported input [1], which is processed now in about 20 second generating ~20k triples. (@ r1328835).
Please confirm performances improvement.
[1] http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259427#comment-13259427 ]
Hudson commented on ANY23-77:
-----------------------------
Integrated in Any23-trunk #179 (See [https://builds.apache.org/job/Any23-trunk/179/])
Replaced node nesting detection logic with higher performance code. This commit is related to issue #ANY23-77. (Revision 1328835)
Result = UNSTABLE
mostarda :
Files :
* /incubator/any23/trunk/core/src/main/java/org/apache/any23/extractor/ExtractionContext.java
* /incubator/any23/trunk/core/src/main/java/org/apache/any23/extractor/microdata/MicrodataParser.java
* /incubator/any23/trunk/core/src/main/java/org/apache/any23/writer/RDFWriterTripleHandler.java
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Simone Tripodi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simone Tripodi updated ANY23-77:
--------------------------------
Component/s: service
Affects Version/s: 0.7.0
Fix Version/s: 0.7.0
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Components: service
> Affects Versions: 0.7.0
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
> Fix For: 0.7.0
>
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ANY23-77) Facing a infinite loop problem in
version 0.6.1 - Verify
Posted by "Michele Mostarda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ANY23-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258848#comment-13258848 ]
Michele Mostarda commented on ANY23-77:
---------------------------------------
It seems your issue is related to https://issues.apache.org/jira/browse/ANY23-76, let's try solve it first.
Best.
Mic
> Facing a infinite loop problem in version 0.6.1 - Verify
> --------------------------------------------------------
>
> Key: ANY23-77
> URL: https://issues.apache.org/jira/browse/ANY23-77
> Project: Apache Any23
> Issue Type: Bug
> Reporter: Michele Mostarda
> Assignee: Michele Mostarda
>
> The code to reproduce the bug is here (Client is Jersey http client, but thats just a detail, the problem lies in URL resource: http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml
> Java Code:
> Client c = Client.create();
> System.out.println("Downloading file.");
> InputStream in = c.resource("http://lod.openlinksw.com/sparql?query=define%20sql%3Adescribe-mode%20%22LOD%22%20%20DESCRIBE%20%3Chttp%3A%2F%2Fyago-knowledge.org%2Fresource%2FBerlin%3E&output=text%2Fhtml").get(InputStream.class);
> FileOutputStream out = null;
> File f = new File("urlResource");
> try {
> out = new FileOutputStream(f);
> IOUtils.copy(in, out);
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> IOUtils.closeQuietly(in);
> IOUtils.closeQuietly(out);
> }
> System.out.println("File downloaded.");
> System.out.println("Starting extraction.");
> FileDocumentSource doc = new FileDocumentSource(f);
> TurtleWriter tw = new TurtleWriter(System.out);
> Any23 ext = new Any23();
> try {
> ext.extract(doc, tw);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("Extraction done.");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira