You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by sushil_kb <ba...@gmail.com> on 2009/10/28 01:39:58 UTC
Re: Creating Vectors from Text
I am having the same problem as Allan. I checked out mahout from trunk and
tried to create term frequency vector from a lucene index and ran into
this..
09/10/27 17:36:10 INFO lucene.Driver: Output File:
/Users/shoeseal/DATA/luc2tvec.out
09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
Exception in thread "main" java.lang.NullPointerException
at
org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
at
org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
at
org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
I am running this from Eclipse (snow leopard with JDK 6), on an index that
has field with stored term vectors..
my input parameters for Driver are:
--dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
--field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
Luke shows the following info on the fields I am using:
id_field is indexed, stored, omit norms
field_with_TV is indexed, tokenized, stored, term vector
I can run the test LuceneIterableTest fine but when I run the Driver on my
index I get into trouble. Any possible reasons for this behavior besides not
having an index field with stored term vector?
Thanks.
- sushil
Grant Ingersoll-6 wrote:
>
>
> On Jul 2, 2009, at 12:09 PM, Allan Roberto Avendano Sudario wrote:
>
>> Regards,
>> This is the entire exception message:
>>
>>
>> java -cp $JAVACLASSPATH org.apache.mahout.utils.vectors.Driver --dir
>> /home/hadoop/Desktop/<urls>/index --field content --dictOut
>> /home/hadoop/Desktop/dictionary/dict.txt --output
>> /home/hadoop/Desktop/dictionary/out.txt --max 50 --norm 2
>>
>>
>> 09/07/02 09:35:47 INFO vectors.Driver: Output File:
>> /home/hadoop/Desktop/dictionary/out.txt
>> 09/07/02 09:35:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 09/07/02 09:35:47 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized
>> native-zlib library
>> 09/07/02 09:35:47 INFO compress.CodecPool: Got brand-new compressor
>> Exception in thread "main" java.lang.NullPointerException
>> at
>> org.apache.mahout.utils.vectors.lucene.LuceneIteratable
>> $TDIterator.next(LuceneIteratable.java:111)
>> at
>> org.apache.mahout.utils.vectors.lucene.LuceneIteratable
>> $TDIterator.next(LuceneIteratable.java:82)
>> at
>> org
>> .apache
>> .mahout
>> .utils
>> .vectors
>> .io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:25)
>> at org.apache.mahout.utils.vectors.Driver.main(Driver.java:204)
>>
>>
>> Well, I used a nutch crawl index, is that correct? mmm... I have
>> change to
>> contenc field, but nothing happened.
>> Possibly the nutch crawl doesn´t have Term Vector indexed.
>
> This would be my guess. A small edit to Nutch code would probably
> allow it. Just find where it creates a new Field and add in the TV
> stuff.
>
--
View this message in context: http://www.nabble.com/Creating-Vectors-from-Text-tp24298643p26087537.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Creating Vectors from Text
Posted by sushil_kb <ba...@gmail.com>.
I created a ticket and attached the patch in Jira
https://issues.apache.org/jira/browse/MAHOUT-191
- sushil
Isabel Drost-4 wrote:
>
> On Tue sushil_kb <ba...@gmail.com> wrote:
>
>> It seems that the problem is because that not all the documents in my
>> index has the field that I am using to get term vectors from. I made
>> the following changes to make this work
>
> Would you please be so kind as to open a jira ticket and attach your
> changes as patch?
>
> Thanks,
> Isabel
>
>
--
View this message in context: http://www.nabble.com/Creating-Vectors-from-Text-tp24298643p26090851.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Creating Vectors from Text
Posted by Isabel Drost <is...@apache.org>.
On Tue sushil_kb <ba...@gmail.com> wrote:
> It seems that the problem is because that not all the documents in my
> index has the field that I am using to get term vectors from. I made
> the following changes to make this work
Would you please be so kind as to open a jira ticket and attach your
changes as patch?
Thanks,
Isabel
Re: Creating Vectors from Text
Posted by sushil_kb <ba...@gmail.com>.
It seems that the problem is because that not all the documents in my index
has the field that I am using to get term vectors from. I made the following
changes to make this work, but I am not sure if thats the right way. I
wanted to get this work to run the LDA topic modeling using the output from
the Driver.
Index:
utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java
===================================================================
---
utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java
(revision 830343)
+++
utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java
(working copy)
@@ -42,7 +42,7 @@
break;
}
//point.write(dataOut);
- writer.append(new LongWritable(recNum++), point);
+ if(point!=null) writer.append(new LongWritable(recNum++), point);
}
return recNum;
Index:
utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java
===================================================================
---
utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java
(revision 830343)
+++
utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java
(working copy)
@@ -104,6 +104,10 @@
try {
indexReader.getTermFreqVector(doc, field, mapper);
result = mapper.getVector();
+
+ if (result == null)
+ return null;
+
if (idField != null) {
String id = indexReader.document(doc,
idFieldSelector).get(idField);
result.setName(id);
sushil_kb wrote:
>
> I am having the same problem as Allan. I checked out mahout from trunk and
> tried to create term frequency vector from a lucene index and ran into
> this..
>
> 09/10/27 17:36:10 INFO lucene.Driver: Output File:
> /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at
> org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at
> org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
>
> I am running this from Eclipse (snow leopard with JDK 6), on an index that
> has field with stored term vectors..
>
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight
> tf
>
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
>
> I can run the test LuceneIterableTest fine but when I run the Driver on my
> index I get into trouble. Any possible reasons for this behavior besides
> not having an index field with stored term vector?
>
> Thanks.
> - sushil
>
>
>
>
> Grant Ingersoll-6 wrote:
>>
>>
>> On Jul 2, 2009, at 12:09 PM, Allan Roberto Avendano Sudario wrote:
>>
>>> Regards,
>>> This is the entire exception message:
>>>
>>>
>>> java -cp $JAVACLASSPATH org.apache.mahout.utils.vectors.Driver --dir
>>> /home/hadoop/Desktop/<urls>/index --field content --dictOut
>>> /home/hadoop/Desktop/dictionary/dict.txt --output
>>> /home/hadoop/Desktop/dictionary/out.txt --max 50 --norm 2
>>>
>>>
>>> 09/07/02 09:35:47 INFO vectors.Driver: Output File:
>>> /home/hadoop/Desktop/dictionary/out.txt
>>> 09/07/02 09:35:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>> 09/07/02 09:35:47 INFO zlib.ZlibFactory: Successfully loaded &
>>> initialized
>>> native-zlib library
>>> 09/07/02 09:35:47 INFO compress.CodecPool: Got brand-new compressor
>>> Exception in thread "main" java.lang.NullPointerException
>>> at
>>> org.apache.mahout.utils.vectors.lucene.LuceneIteratable
>>> $TDIterator.next(LuceneIteratable.java:111)
>>> at
>>> org.apache.mahout.utils.vectors.lucene.LuceneIteratable
>>> $TDIterator.next(LuceneIteratable.java:82)
>>> at
>>> org
>>> .apache
>>> .mahout
>>> .utils
>>> .vectors
>>> .io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:25)
>>> at org.apache.mahout.utils.vectors.Driver.main(Driver.java:204)
>>>
>>>
>>> Well, I used a nutch crawl index, is that correct? mmm... I have
>>> change to
>>> contenc field, but nothing happened.
>>> Possibly the nutch crawl doesn´t have Term Vector indexed.
>>
>> This would be my guess. A small edit to Nutch code would probably
>> allow it. Just find where it creates a new Field and add in the TV
>> stuff.
>>
>
>
--
View this message in context: http://www.nabble.com/Creating-Vectors-from-Text-tp24298643p26087765.html
Sent from the Mahout User List mailing list archive at Nabble.com.