[jira] Issue Comment Edited: (LUCENE-2731) HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK 1.4)

["Uwe Schindler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926689#action_12926689 ] 

Uwe Schindler edited comment on LUCENE-2731 at 10/31/10 6:05 AM:
-----------------------------------------------------------------

By the way, the whole loading in the TokenFilter is broken! XML files should never be load by a Reader, always by InputStream. The charset detection is part of the XML spec and the default is UTF-8 if not overridden in the XML file. The HyphenationCompoundWordTokenFilter supplies ISO-8859-1 as fixed for the reader, so it is never possible to load other xml files with different charsets. This is another issue, I will open for 3.x and trunk (as it needs API change).

      was (Author: thetaphi):
    By the way, the whole loading in the TokenFilter is broken! XML files should never be load by a Reaer, always by InputStream. The charset detection is part of the XML spec and the default is UTF-8 if not overridden in the XML file. The HyphenationCompoundWordTokenFilter supplies ISO-8859-1 as fixed for the reader, so it is never possible to load other xml files with different charsets. This is another issue, I will open for 3.x and trunk (as it needs API change).
  
> HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK 1.4)
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-2731
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2731
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/analyzers
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 2.9.4
>
>         Attachments: LUCENE-2731.patch
>
>
> HyphenationCompoundWordTokenFilter loads the DTD in its XML parser from memory by supplying EntityResolver. In Java 1.4 (affects Lucene 2.9, but also later versions if not Apache Xerces is used as XML parser) this does not work, because Cromson does not even ask the entity resolver, if no base URI is known. As the hyphenation file is loaded from Reader/InputStream no base URI is known. Crimson needs at least a non-null systemId to proceed.
> This patch (Lucene 2.9 only)  fakes this by supplying a fake systemId to the InputSource.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org