You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by kratoras <kr...@hotmail.com> on 2008/02/18 13:08:15 UTC
Problem using Lucene on Ubuntu
Hello!
I ve written a sample application which indexes documents written in Greek
using the GreekAnalyzer and search these documents with both greek and
english words. Though on Windows the searching returns correct results, if i
try it on Ubuntu the searching does not return any results for any greek
word but it continues to search for english words. Could it be a problem of
fonts in my ubuntu or what?Greek fonts are installed though.
Thanks for any answers in advance
--
View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15543843.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by Grant Ingersoll <gs...@apache.org>.
Good point Jan!
On Feb 18, 2008, at 9:00 AM, Jan Peter Stotz wrote:
> Grant Ingersoll wrote:
>
>> Note: ENCODING is whatever encoding the file is in, as in "UTF-8",
>> if that is what your files are in.
>
> I think there is a misunderstanding, the WordExtractor extracts text
> from MS Word (.doc) files. Those files are binary and therefore does
> not have an encoding.
> I would print out the extracted text into a plain text files and
> compare if there are differences between the file generated on
> Windows and Linux/Ubuntu. This allows to determine if this is a
> WordExtractor or a Lucene problem.
>
> Jan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by Jan Peter Stotz <jp...@gmx.de>.
Grant Ingersoll wrote:
> Note: ENCODING is whatever encoding the file is in, as in "UTF-8", if
> that is what your files are in.
I think there is a misunderstanding, the WordExtractor extracts text
from MS Word (.doc) files. Those files are binary and therefore does not
have an encoding.
I would print out the extracted text into a plain text files and compare
if there are differences between the file generated on Windows and
Linux/Ubuntu. This allows to determine if this is a WordExtractor or a
Lucene problem.
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by Grant Ingersoll <gs...@apache.org>.
Not sure about WordExtractor, does it also take a Reader? I would try:
Reader input = new InputStreamReader(new FileInputStream(file),
"ENCODING");
WordExtractor extractor = new WordExtractor(input);
content = extractor.getText();
Note: ENCODING is whatever encoding the file is in, as in "UTF-8", if
that is what your files are in. If you don't know the encoding, you
will need to add in some type of character encoding detection tool.
Search the web for that, as I know there are some out there (I don't
know any names off hand).
Bottom line, it sounds like you need to figure out how to load your
files based on their encoding. That problem is not really core to
Lucene, but you should be able to search the archives here to find
others with similar questions.
-Grant
On Feb 18, 2008, at 8:13 AM, kratoras wrote:
>
> No problem about the misunderstanding.
> I am using
>
> InputStream input =new URL ( "file:///"+file.getAbsolutePath()
> ).openStream ();
> WordExtractor extractor = new WordExtractor(input);
> content=extractor.getText();
>
> where the wordextractor is
> org.apache.poi.hwpf.extractor.WordExtractor;
>
> The wordextractor takes an inputstream as an arguement. Should i
> determine
> the encoding of the inputstream and how?
> --
> View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15545082.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by kratoras <kr...@hotmail.com>.
No problem about the misunderstanding.
I am using
InputStream input =new URL ( "file:///"+file.getAbsolutePath()
).openStream ();
WordExtractor extractor = new WordExtractor(input);
content=extractor.getText();
where the wordextractor is org.apache.poi.hwpf.extractor.WordExtractor;
The wordextractor takes an inputstream as an arguement. Should i determine
the encoding of the inputstream and how?
--
View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15545082.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by Grant Ingersoll <gs...@apache.org>.
How are you loading the document into the content variable below? My
guess is still that you have different locales on Windows and Ubuntu.
(Btw, sorry about the java-user comment. I should wake up before
sending responses. For some reason I thought the email was sent to
java-dev)
-Grant
On Feb 18, 2008, at 7:44 AM, kratoras wrote:
>
> Actually what i figured out just now is that the problem is on the
> indexing
> part. A document with a 15MB size is transformed in a 23MB index
> which is
> not normal since on windows for the same document the index is 3MB.
> For the
> indexing i use:
> writer = new IndexWriter(index, new GreekAnalyzer(), !index.exists());
> and to add documents:
> doc.add(new
> Field("contents",content,Field.Store.YES,Field.Index.TOKENIZED));
>
> where "content" is a string with the content of the document. Should i
> convert this string to UTF-8 using getBytes before i write it to the
> index??
>
> --
> View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15544612.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by kratoras <kr...@hotmail.com>.
Actually what i figured out just now is that the problem is on the indexing
part. A document with a 15MB size is transformed in a 23MB index which is
not normal since on windows for the same document the index is 3MB. For the
indexing i use:
writer = new IndexWriter(index, new GreekAnalyzer(), !index.exists());
and to add documents:
doc.add(new
Field("contents",content,Field.Store.YES,Field.Index.TOKENIZED));
where "content" is a string with the content of the document. Should i
convert this string to UTF-8 using getBytes before i write it to the index??
--
View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15544612.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using Lucene on Ubuntu
Posted by Grant Ingersoll <gs...@apache.org>.
This question is best asked on java-user. However, my guess is that
it is related to your Locale and that you need to set the character
encoding to Greek on Ubuntu when reading in your files.
Something like: Reader reader = new InputStreamReader(new
FileInputStream(file), "GREEK Char Encoding here");
-Grant
On Feb 18, 2008, at 7:08 AM, kratoras wrote:
>
> Hello!
> I ve written a sample application which indexes documents written in
> Greek
> using the GreekAnalyzer and search these documents with both greek and
> english words. Though on Windows the searching returns correct
> results, if i
> try it on Ubuntu the searching does not return any results for any
> greek
> word but it continues to search for english words. Could it be a
> problem of
> fonts in my ubuntu or what?Greek fonts are installed though.
>
> Thanks for any answers in advance
> --
> View this message in context: http://www.nabble.com/Problem-using-Lucene-on-Ubuntu-tp15543843p15543843.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org