You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2009/07/06 21:35:14 UTC
[jira] Commented: (LUCENE-1704) org.apache.lucene.ant.HtmlDocument
added Tidy config file passthrough availability
[ https://issues.apache.org/jira/browse/LUCENE-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727723#action_12727723 ]
Michael McCandless commented on LUCENE-1704:
--------------------------------------------
There is a preview button (that small blue computer-like icon to the left of the yellow question mark help icon), and, you can use \{code} to wrap code so Jira doesn't format it.
> org.apache.lucene.ant.HtmlDocument added Tidy config file passthrough availability
> ----------------------------------------------------------------------------------
>
> Key: LUCENE-1704
> URL: https://issues.apache.org/jira/browse/LUCENE-1704
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/*
> Affects Versions: 2.4.1
> Reporter: Keith Sprochi
> Priority: Trivial
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Parsing HTML documents using the org.apache.lucene.ant.HtmlDocument.Document method resulted in many error messages such as this:
> line 152 column 725 - Error: <as-html> is not recognized!
> This document has errors that must be fixed before
> using HTML Tidy to generate a tidied up version.
> The solution is to configure Tidy to accept these abnormal tags by adding the tag name to the "new-inline-tags" option in the Tidy config file (or the command line which does not make sense in this context), like so:
> new-inline-tags: as-html
> Tidy needs to know where the configuration file is, so a new constructor and Document method can be added. Here is the code:
> /**
> * Constructs an <code>HtmlDocument</code> from a {@link
> * java.io.File}.
> *
> *@param file the <code>File</code> containing the
> * HTML to parse
> *@param tidyConfigFile the <code>String</code> containing
> * the full path to the Tidy config file
> *@exception IOException if an I/O exception occurs
> */
> public HtmlDocument(File file, String tidyConfigFile) throws IOException {
> Tidy tidy = new Tidy();
> tidy.setConfigurationFromFile(tidyConfigFile);
> tidy.setQuiet(true);
> tidy.setShowWarnings(false);
> org.w3c.dom.Document root =
> tidy.parseDOM(new FileInputStream(file), null);
> rawDoc = root.getDocumentElement();
> }
> /**
> * Creates a Lucene <code>Document</code> from a {@link
> * java.io.File}.
> *
> *@param file
> *@param tidyConfigFile the full path to the Tidy config file
> *@exception IOException
> */
> public static org.apache.lucene.document.Document
> Document(File file, String tidyConfigFile) throws IOException {
> HtmlDocument htmlDoc = new HtmlDocument(file, tidyConfigFile);
> org.apache.lucene.document.Document luceneDoc = new org.apache.lucene.document.Document();
> luceneDoc.add(new Field("title", htmlDoc.getTitle(), Field.Store.YES, Field.Index.ANALYZED));
> luceneDoc.add(new Field("contents", htmlDoc.getBody(), Field.Store.YES, Field.Index.ANALYZED));
> String contents = null;
> BufferedReader br =
> new BufferedReader(new FileReader(file));
> StringWriter sw = new StringWriter();
> String line = br.readLine();
> while (line != null) {
> sw.write(line);
> line = br.readLine();
> }
> br.close();
> contents = sw.toString();
> sw.close();
> luceneDoc.add(new Field("rawcontents", contents, Field.Store.YES, Field.Index.NO));
> return luceneDoc;
> }
> I am using this now and it is working fine. The configuration file is being passed to Tidy and now I am able to index thousands of HTML pages with no more Tidy tag errors.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org