You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Patrick Oliver Glauner <pa...@cern.ch> on 2012/09/25 18:43:11 UTC
Indexing in Solr: invalid UTF-8
Hello
We use Solr 3.1 and Jetty to index previously extracted fulltexts from PDFs, DOC etc. Our indexing script is written in Python 2.4 using solrpy:
[...]
text = remove_control_characters(text) # except \r, \t, \n
utext = unicode(text, 'utf-8')
SOLR_CONNECTION.add(id=recid, fulltext=utext)
[...]
But for some fulltexts we still get exceptions like:
* [was class java.io.CharConversionException] Invalid UTF-8 character 0xd835(a surrogate character) at char #1144, byte #127)
* [was class java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #1427640, byte #1564649)
* ....
Why does this exceptions still occur? How can I avoid these exceptions? I hoped that utext = unicode(text, 'utf-8') was enough.
Thanks
Patrick
FYI, the fulltext field definition is:
<field name="fulltext" type="invenioText" indexed="true" stored="false" multiValued="true"/>
where
<fieldType name="invenioText" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="1"
preserveOriginal="1"
splitOnNumerics ="1"
splitOnCaseChange ="1"
stemEnglishPossessive="1"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"/>
<filter class="solr.LengthFilterFactory" min="2" max="99"/>
</analyzer>
[...]
</fieldType>
--
Patrick GLAUNER [patrick.oliver.glauner@cern.ch]
CERN
Information Technology Department
CH-1211 Geneva 23
Re: Indexing in Solr: invalid UTF-8
Posted by Gora Mohanty <go...@mimirtech.com>.
On 9 October 2012 17:42, Patrick Oliver Glauner
<pa...@cern.ch> wrote:
> Hello everybody
>
> Meanwhile, I checked this issue in detail: we use pdftotext to extract text from our PDFs (<http://cds.cern.ch/>). Some generated text files contain \uFFFF and \uD835.
>
> unicode(text, 'utf-8') does not throw any exception for these texts. Subsequently, Solr throws an exception when these are sent to the indexer.
Off-topic, but this is because the Unicode escape sequence
'\uxxxx' is not being interpreted here. You have to explicitly
do that. Here is an example with '\u2018', the opening
quote (I did not have a font which covered '\ud835'). Please
note the difference between:
print unicode('\u2018')
\u2018
and
print unicode('\u2018').decode('unicode-escape')
‘
Regards,
Gora
RE: Indexing in Solr: invalid UTF-8
Posted by Patrick Oliver Glauner <pa...@cern.ch>.
Hello everybody
Meanwhile, I checked this issue in detail: we use pdftotext to extract text from our PDFs (<http://cds.cern.ch/>). Some generated text files contain \uFFFF and \uD835.
unicode(text, 'utf-8') does not throw any exception for these texts. Subsequently, Solr throws an exception when these are sent to the indexer.
Therefore, I wrote a little function to remove them after the unicode() call:
+def remove_invalid_solr_characters(utext):
+ for char in CFG_SOLR_INVALID_CHAR_REPLACEMENTS:
+ try:
+ utext = utext.replace(char, CFG_SOLR_INVALID_CHAR_REPLACEMENTS[char])
+ except:
+ pass
+ return utext
with:
+CFG_SOLR_INVALID_CHAR_REPLACEMENTS = {
+ u'\uFFFF' : u"",
+ u'\uD835' : u""
+}
This works well in our production environment.
Cheers, Patrick
________________________________________
From: Patrick Oliver Glauner [patrick.oliver.glauner@cern.ch]
Sent: Friday, September 28, 2012 10:36 AM
To: solr-user@lucene.apache.org
Subject: RE: Indexing in Solr: invalid UTF-8
Thank you. I will check our textification process and see how to improve it.
Patrick
________________________________________
From: Michael McCandless [lucene@mikemccandless.com]
Sent: Wednesday, September 26, 2012 5:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Indexing in Solr: invalid UTF-8
Python's unicode function takes an optional (keyword) "errors"
argument, telling it what to do when an invalid UTF8 byte sequence is
seen.
The default (errors='strict') is to throw the exceptions you're
seeing. But you can also pass errors='replace' or errors='ignore'.
See http://docs.python.org/howto/unicode.html for details ...
However, I agree with Robert: you should dig into why whatever process
you used to extract the full text from your binary documents is
producing invalid UTF-8 ... something is wrong with that process.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Sep 25, 2012 at 10:44 PM, Robert Muir <rc...@gmail.com> wrote:
> On Tue, Sep 25, 2012 at 2:02 PM, Patrick Oliver Glauner
> <pa...@cern.ch> wrote:
>> Hi
>> Thanks. But I see that 0xd835 is missing in this list (see my exceptions).
>>
>> What's the best way to get rid of all of them in Python? I am new to unicode in Python but I am sure that this use case is quite frequent.
>>
>
> I don't really know python either: so I could be wrong here but are
> you just taking these binary .PDF and .DOC files and treating them as
> UTF-8 text and sending them to Solr?
>
> If so, I don't think that will work very well. Maybe instead try
> parsing these binary files with something like Tika to get at the
> actual content and send that? (it seems some people have developed
> python integration for this, e.g.
> http://redmine.djity.net/projects/pythontika/wiki)
RE: Indexing in Solr: invalid UTF-8
Posted by Patrick Oliver Glauner <pa...@cern.ch>.
Thank you. I will check our textification process and see how to improve it.
Patrick
________________________________________
From: Michael McCandless [lucene@mikemccandless.com]
Sent: Wednesday, September 26, 2012 5:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Indexing in Solr: invalid UTF-8
Python's unicode function takes an optional (keyword) "errors"
argument, telling it what to do when an invalid UTF8 byte sequence is
seen.
The default (errors='strict') is to throw the exceptions you're
seeing. But you can also pass errors='replace' or errors='ignore'.
See http://docs.python.org/howto/unicode.html for details ...
However, I agree with Robert: you should dig into why whatever process
you used to extract the full text from your binary documents is
producing invalid UTF-8 ... something is wrong with that process.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Sep 25, 2012 at 10:44 PM, Robert Muir <rc...@gmail.com> wrote:
> On Tue, Sep 25, 2012 at 2:02 PM, Patrick Oliver Glauner
> <pa...@cern.ch> wrote:
>> Hi
>> Thanks. But I see that 0xd835 is missing in this list (see my exceptions).
>>
>> What's the best way to get rid of all of them in Python? I am new to unicode in Python but I am sure that this use case is quite frequent.
>>
>
> I don't really know python either: so I could be wrong here but are
> you just taking these binary .PDF and .DOC files and treating them as
> UTF-8 text and sending them to Solr?
>
> If so, I don't think that will work very well. Maybe instead try
> parsing these binary files with something like Tika to get at the
> actual content and send that? (it seems some people have developed
> python integration for this, e.g.
> http://redmine.djity.net/projects/pythontika/wiki)
Re: Indexing in Solr: invalid UTF-8
Posted by Michael McCandless <lu...@mikemccandless.com>.
Python's unicode function takes an optional (keyword) "errors"
argument, telling it what to do when an invalid UTF8 byte sequence is
seen.
The default (errors='strict') is to throw the exceptions you're
seeing. But you can also pass errors='replace' or errors='ignore'.
See http://docs.python.org/howto/unicode.html for details ...
However, I agree with Robert: you should dig into why whatever process
you used to extract the full text from your binary documents is
producing invalid UTF-8 ... something is wrong with that process.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Sep 25, 2012 at 10:44 PM, Robert Muir <rc...@gmail.com> wrote:
> On Tue, Sep 25, 2012 at 2:02 PM, Patrick Oliver Glauner
> <pa...@cern.ch> wrote:
>> Hi
>> Thanks. But I see that 0xd835 is missing in this list (see my exceptions).
>>
>> What's the best way to get rid of all of them in Python? I am new to unicode in Python but I am sure that this use case is quite frequent.
>>
>
> I don't really know python either: so I could be wrong here but are
> you just taking these binary .PDF and .DOC files and treating them as
> UTF-8 text and sending them to Solr?
>
> If so, I don't think that will work very well. Maybe instead try
> parsing these binary files with something like Tika to get at the
> actual content and send that? (it seems some people have developed
> python integration for this, e.g.
> http://redmine.djity.net/projects/pythontika/wiki)
Re: Indexing in Solr: invalid UTF-8
Posted by Robert Muir <rc...@gmail.com>.
On Tue, Sep 25, 2012 at 2:02 PM, Patrick Oliver Glauner
<pa...@cern.ch> wrote:
> Hi
> Thanks. But I see that 0xd835 is missing in this list (see my exceptions).
>
> What's the best way to get rid of all of them in Python? I am new to unicode in Python but I am sure that this use case is quite frequent.
>
I don't really know python either: so I could be wrong here but are
you just taking these binary .PDF and .DOC files and treating them as
UTF-8 text and sending them to Solr?
If so, I don't think that will work very well. Maybe instead try
parsing these binary files with something like Tika to get at the
actual content and send that? (it seems some people have developed
python integration for this, e.g.
http://redmine.djity.net/projects/pythontika/wiki)
RE: Indexing in Solr: invalid UTF-8
Posted by Patrick Oliver Glauner <pa...@cern.ch>.
Hi
Thanks. But I see that 0xd835 is missing in this list (see my exceptions).
What's the best way to get rid of all of them in Python? I am new to unicode in Python but I am sure that this use case is quite frequent.
Patrick
________________________________________
From: Markus Jelsma [markus.jelsma@openindex.io]
Sent: Tuesday, September 25, 2012 7:24 PM
To: solr-user@lucene.apache.org; Patrick Oliver Glauner
Subject: RE: Indexing in Solr: invalid UTF-8
Hi - you need to get rid of all non-character code points.
http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:Noncharacter_Code_Point=True:]
-----Original message-----
> From:Patrick Oliver Glauner <pa...@cern.ch>
> Sent: Tue 25-Sep-2012 18:47
> To: solr-user@lucene.apache.org
> Subject: Indexing in Solr: invalid UTF-8
>
> Hello
>
> We use Solr 3.1 and Jetty to index previously extracted fulltexts from PDFs, DOC etc. Our indexing script is written in Python 2.4 using solrpy:
>
> [...]
> text = remove_control_characters(text) # except \r, \t, \n
> utext = unicode(text, 'utf-8')
> SOLR_CONNECTION.add(id=recid, fulltext=utext)
> [...]
>
> But for some fulltexts we still get exceptions like:
>
> * [was class java.io.CharConversionException] Invalid UTF-8 character 0xd835(a surrogate character) at char #1144, byte #127)
> * [was class java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #1427640, byte #1564649)
> * ....
>
> Why does this exceptions still occur? How can I avoid these exceptions? I hoped that utext = unicode(text, 'utf-8') was enough.
>
> Thanks
> Patrick
>
>
> FYI, the fulltext field definition is:
>
> <field name="fulltext" type="invenioText" indexed="true" stored="false" multiValued="true"/>
>
> where
>
> <fieldType name="invenioText" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1"
> generateNumberParts="1"
> catenateWords="0"
> catenateNumbers="0"
> catenateAll="1"
> preserveOriginal="1"
> splitOnNumerics ="1"
> splitOnCaseChange ="1"
> stemEnglishPossessive="1"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"/>
> <filter class="solr.LengthFilterFactory" min="2" max="99"/>
>
> </analyzer>
> [...]
> </fieldType>
>
>
> --
> Patrick GLAUNER [patrick.oliver.glauner@cern.ch]
>
> CERN
> Information Technology Department
> CH-1211 Geneva 23
>
RE: Indexing in Solr: invalid UTF-8
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - you need to get rid of all non-character code points.
http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:Noncharacter_Code_Point=True:]
-----Original message-----
> From:Patrick Oliver Glauner <pa...@cern.ch>
> Sent: Tue 25-Sep-2012 18:47
> To: solr-user@lucene.apache.org
> Subject: Indexing in Solr: invalid UTF-8
>
> Hello
>
> We use Solr 3.1 and Jetty to index previously extracted fulltexts from PDFs, DOC etc. Our indexing script is written in Python 2.4 using solrpy:
>
> [...]
> text = remove_control_characters(text) # except \r, \t, \n
> utext = unicode(text, 'utf-8')
> SOLR_CONNECTION.add(id=recid, fulltext=utext)
> [...]
>
> But for some fulltexts we still get exceptions like:
>
> * [was class java.io.CharConversionException] Invalid UTF-8 character 0xd835(a surrogate character) at char #1144, byte #127)
> * [was class java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #1427640, byte #1564649)
> * ....
>
> Why does this exceptions still occur? How can I avoid these exceptions? I hoped that utext = unicode(text, 'utf-8') was enough.
>
> Thanks
> Patrick
>
>
> FYI, the fulltext field definition is:
>
> <field name="fulltext" type="invenioText" indexed="true" stored="false" multiValued="true"/>
>
> where
>
> <fieldType name="invenioText" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1"
> generateNumberParts="1"
> catenateWords="0"
> catenateNumbers="0"
> catenateAll="1"
> preserveOriginal="1"
> splitOnNumerics ="1"
> splitOnCaseChange ="1"
> stemEnglishPossessive="1"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"/>
> <filter class="solr.LengthFilterFactory" min="2" max="99"/>
>
> </analyzer>
> [...]
> </fieldType>
>
>
> --
> Patrick GLAUNER [patrick.oliver.glauner@cern.ch]
>
> CERN
> Information Technology Department
> CH-1211 Geneva 23
>