You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2010/10/31 13:13:23 UTC
[jira] Created: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
----------------------------------------------------------------------------------------------------------
Key: LUCENE-2732
URL: https://issues.apache.org/jira/browse/LUCENE-2732
Project: Lucene - Java
Issue Type: Bug
Reporter: Uwe Schindler
Assignee: Uwe Schindler
Fix For: 2.9.4, 3.0.3, 3.1, 4.0
As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2732:
----------------------------------
Description:
As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
was:
As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility (deprecated).
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2732:
----------------------------------
Attachment: LUCENE-2732.patch
Patch that fixes XML parsing, to apply first move the hyphenation.dtd to the src/resource/ folder.
This patch also removed the hardcoded DTD from the parser and moves it to the resources folder (loaded by classloader). Solr is fixed to use the InputSource API, but it shozuld really use an URL, so ResourceLoader in Solr should be fixed to also supply URLs like ClassLoader!
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2732.patch
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-2732.
-----------------------------------
Resolution: Fixed
Committed 3.0 revision: 1029374
Committed 2.9 revision: 1029375
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2732.patch, LUCENE-2732.patch
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926708#action_12926708 ]
Uwe Schindler commented on LUCENE-2732:
---------------------------------------
Committed truk revision: 1029345
Now backporting...
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2732.patch, LUCENE-2732.patch
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2732:
----------------------------------
Attachment: LUCENE-2732.patch
Updated patch, uses Locale.ENGLISH as noted by Robert. Also leaves the DTD in place (Solr+Lucene tests, but it is never parsed, only if you would view the xml in the tests in your favourite XML reader)
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2732.patch, LUCENE-2732.patch
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2732) Fix charset problems in XML loading
in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926712#action_12926712 ]
Uwe Schindler commented on LUCENE-2732:
---------------------------------------
Committed branch 3.x revision: 1029350
Backporting bugfix only to 3.0/2.9!
> Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2732
> URL: https://issues.apache.org/jira/browse/LUCENE-2732
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2732.patch, LUCENE-2732.patch
>
>
> As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.
> This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org