Lucene Indexer Encoding problem

[svirid <te...@gmail.com>]

Good day guys,

hope u can help me. I am trying to index French and Russian documents with
Lucene and have no luck. I am new in JAVA so basically I really need your
help. 

I was able to get text from pdfs, when I save it its all fine I can clearly
see russian charachters in txt file but when I add it to the Index its all
??? or other garbage.

Here is what I do: 

I first use PDF box to extract text. 

[CODE]
textFile = "c:/java/faq.txt";
pdfFile = "c:/java/faq.pdf"; 

//FIRST I AM GETTING TEXT FROM PDF
document = PDDocument.load( pdfFile );

output = new OutputStreamWriter  ( new FileOutputStream  ( textFile ),
"UTF-8" );	    
PDFTextStripper stripper = new PDFTextStripper();
stripper.setStartPage( 1 );
stripper.setEndPage( 20 );

//THIS SAVES TEXT INTO THE TXT FILE, TXT FILE COMPLETELY FINE
stripper.writeText(document, output);

//BUT WHEN I GET TEXT LIKE THAT TO ADD TO THE INDEX
textData = stripper.getText(document);


Analyzer analyzer = new StandardAnalyzer();        
Directory directory = FSDirectory.getDirectory("c:/java/collection");
IndexWriter iwriter = new IndexWriter(directory, analyzer, new
IndexWriter.MaxFieldLength(250));
Document doc = new Document();
        
doc.add(new Field("fieldname", textData, Field.Store.YES,
Field.Index.NOT_ANALYZED));
iwriter.addDocument(doc);
iwriter.optimize();
iwriter.close();
[/CODE]

This code above properly saves extracted text to the txt file, whioch I dotn
really need. What I want is to get text and add it to the Index right away.
When I open index files in notepad I can see garbage instead of russian
characters. 

Please help. Thank you
-- 
View this message in context: http://www.nabble.com/Lucene-Indexer-Encoding-problem-tp19959504p19959504.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: Lucene Indexer Encoding problem

[Chris Hostetter <ho...@fucit.org>]

: //BUT WHEN I GET TEXT LIKE THAT TO ADD TO THE INDEX
: textData = stripper.getText(document);

have you looked at the String in textData to make sure it's what you 
expect?

: This code above properly saves extracted text to the txt file, whioch I dotn
: really need. What I want is to get text and add it to the Index right away.
: When I open index files in notepad I can see garbage instead of russian
: characters. 

you can't "open index files in notepad" ... they are binary files, not 
text files.  if you want to see what actaul terms are index, you can use 
tools like Luke to inspect that data.

BTW...

http://people.apache.org/~hossman/#java-dev
Please Use "java-user@lucene" Not "java-dev@lucene"

Your question is better suited for the java-user@lucene mailing list ...
not the java-dev@lucene list.  java-dev is for discussing development of
the internals of the Lucene Java library ... it is *not* the appropriate
place to ask questions about how to use the Lucene Java library when
developing your own applications.  Please resend your message to
the java-user mailing list, where you are likely to get more/better
responses since that list also has a larger number of subscribers.





-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org