You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (Created) (JIRA)" <ji...@apache.org> on 2012/03/29 18:40:27 UTC
[jira] [Created] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Workaround the XERCES-J bug in Benchmark
----------------------------------------
Key: LUCENE-3937
URL: https://issues.apache.org/jira/browse/LUCENE-3937
Project: Lucene - Java
Issue Type: Bug
Reporter: Uwe Schindler
In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241445#comment-13241445 ]
Michael McCandless commented on LUCENE-3937:
--------------------------------------------
Note: I just run benchmark's conf/extractWikipedia.alg task on the XML export... when XERCESJ-1257 strikes you get this:
{noformat}
...
[java] 936.83 sec --> main Wrote 2801000 line docs
[java] 937.04 sec --> main Wrote 2802000 line docs
[java] 937.27 sec --> main Wrote 2803000 line docs
[java] 937.53 sec --> main Wrote 2804000 line docs
[java] 937.79 sec --> main Wrote 2805000 line docs
[java] 938.04 sec --> main Wrote 2806000 line docs
[java] 938.35 sec --> main Wrote 2807000 line docs
[java] 938.65 sec --> main Wrote 2808000 line docs
[java] 938.88 sec --> main Wrote 2809000 line docs
[java] 939.09 sec --> main Wrote 2810000 line docs
[java] 939.09 sec --> main Wrote 2810000 line docs
[java] Exception in thread "Thread-0" java.lang.RuntimeException: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence.
[java] at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:198)
[java] at java.lang.Thread.run(Thread.java:619)
[java] ####################
[java] ### D O N E !!! ###
[java] Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence.
[java] ####################
[java] at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
[java] at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
[java] at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
[java] at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
[java] at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
[java] at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
[java] at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
[java] at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
[java] at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
[java] at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
[java] at org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:175)
[java] ... 1 more
[java] Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence.
[java] at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown Source)
[java] at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
[java] at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
[java] at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
[java] at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source)
[java] ... 8 more
{noformat}
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241438#comment-13241438 ]
Michael McCandless commented on LUCENE-3937:
--------------------------------------------
LUCENE-1591 is when we first tripped on the XERCESJ-1257 bug... and the bug also happens on enwiki-20110115-pages-articles.xml.bz2 export.
Great idea to workaround Xercesj's bug by using the JVM to decode UTF8, instead of Xercesj...
I'll test this patch now!
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241899#comment-13241899 ]
Steven Rowe commented on LUCENE-3937:
-------------------------------------
bq. I added a changes entry?
Benchmark has its own CHANGES.txt, and there is mention in there of this patched jar, so I thought it appropriate to add an entry there. I didn't think to check for your CHANGES entry. I'll go do that now.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241523#comment-13241523 ]
Uwe Schindler commented on LUCENE-3937:
---------------------------------------
Robert, you know better how to do the problem reporting... I have no idea, I only know it's a nice builder-API :-)
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241534#comment-13241534 ]
Robert Muir commented on LUCENE-3937:
-------------------------------------
I can do it... give me a sec
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3937:
--------------------------------
Attachment: LUCENE-3937-remaining-references.patch
Patch against branch_3x removing remaining references to the patched xercesImpl jar. Also adds benchmark CHANGES entry.
Committing shortly, and then forward porting to trunk.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241447#comment-13241447 ]
Robert Muir commented on LUCENE-3937:
-------------------------------------
I agree this is an awesome idea... maybe the reader should not be passed a charset though,
but a charsetdecoder with REPORT set for onXXXX() methods? This way if the xml is corrumpt
or maybe not actually utf-8 (aren't all wikipedias xmls utf-8?), then you know about it.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3937:
--------------------------------
Attachment: LUCENE-3937.patch
updated (untested) patch with issue # added to the comments, and throwing exception on broken encoding.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3937:
--------------------------------
Attachment: LUCENE-3937-remaining-references.patch
Updated branch_3x patch putting back the xercesImpl mention in {{lucene/NOTICE.txt}}.
Uwe, I looked at your CHANGES entry, and I think the entry I wrote in benchmark CHANGES.txt should still be included there. Can you take a look and tell me if you disagree?
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241896#comment-13241896 ]
Robert Muir commented on LUCENE-3937:
-------------------------------------
wait: i don't think we should remove the licensing information totally?
we still rely on xerces. it should just say 2.9.1 (not patched-hacked version)
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-3937.
-----------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.6
Committed trunk 1307141, 3.x 1307144
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3937:
----------------------------------
Priority: Blocker (was: Major)
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Priority: Blocker
> Attachments: LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241486#comment-13241486 ]
Michael McCandless commented on LUCENE-3937:
--------------------------------------------
OK with this patch the decode of enwiki-20110115 finished!
I agree we should tell the decoder to throw exception on any problems...
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241906#comment-13241906 ]
Steven Rowe commented on LUCENE-3937:
-------------------------------------
{quote}
bq. wait: i don't think we should remove the licensing information totally?
bq. we still rely on xerces. it should just say 2.9.1 (not patched-hacked version)
Right, thanks, I'll put it back and adjust the version.
{quote}
So, I'll put it back and adjust the version in {{lucene/NOTICE.txt}}, but think it should be removed from {{solr/NOTICE.txt}} because it's not actually included in Solr? Here's what's in {{solr/NOTICE.txt}} now:
{noformat}
Includes software from other Apache Software Foundation projects,
including, but not limited to:
[...]
- Xerces (lib/xercesImpl-2.9.1-patched-XERCESJ-1257.jar)
{noformat}
No xercesImpl jar exists under {{solr/lib/}}.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241925#comment-13241925 ]
Steven Rowe commented on LUCENE-3937:
-------------------------------------
Committed the remaining references patch to branch_3x and trunk. Uwe, you can kill the {{benchmark/CHANGES.txt}} entry I added if you don't like it.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reassigned LUCENE-3937:
-------------------------------------
Assignee: Uwe Schindler
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3937:
----------------------------------
Attachment: LUCENE-3937.patch
Simple patch. Mike can you test this (by first replacing with stock released XERCES)?
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Attachments: LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241897#comment-13241897 ]
Uwe Schindler commented on LUCENE-3937:
---------------------------------------
I added a changes entry?
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3937) Workaround the XERCES-J bug in
Benchmark
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241901#comment-13241901 ]
Steven Rowe commented on LUCENE-3937:
-------------------------------------
{quote}
wait: i don't think we should remove the licensing information totally?
we still rely on xerces. it should just say 2.9.1 (not patched-hacked version)
{quote}
Right, thanks, I'll put it back and adjust the version.
> Workaround the XERCES-J bug in Benchmark
> ----------------------------------------
>
> Key: LUCENE-3937
> URL: https://issues.apache.org/jira/browse/LUCENE-3937
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Blocker
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3937-remaining-references.patch, LUCENE-3937.patch, LUCENE-3937.patch
>
>
> In becnhmark we have a patched version of XERCES which is hard to compile from source. When looking at the code part patched and the source of EnwikiContentSource, to simply provide the XML parser a Reader instead of InputStream, so the broken code is not triggered. This assumes, that the XML-file is always UTF-8.... If not it will no longer work (because the XML parser cannot switch encoding, if it only has a Reader).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org