You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Sushil Bajracharya (JIRA)" <ji...@apache.org> on 2009/10/28 09:15:59 UTC
[jira] Created: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
NPE while creating term vectors with an index on a field that does not exist in all the documents
-------------------------------------------------------------------------------------------------
Key: MAHOUT-191
URL: https://issues.apache.org/jira/browse/MAHOUT-191
Project: Mahout
Issue Type: Bug
Affects Versions: 0.3
Environment: mac, snow leopard, eclipse galileo, jdk 6
Reporter: Sushil Bajracharya
(based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
Exception in thread "main" java.lang.NullPointerException
at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
my input parameters for Driver are:
--dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
--field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
Luke shows the following info on the fields I am using:
id_field is indexed, stored, omit norms
field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Shashikant Kore (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shashikant Kore updated MAHOUT-191:
-----------------------------------
Attachment: MAHOUT-191.patch
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Attachments: MAHOUT-191-patch.txt, MAHOUT-191.patch
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Shashikant Kore (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786782#action_12786782 ]
Shashikant Kore commented on MAHOUT-191:
----------------------------------------
Yeah, this can be committed.
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Assignee: Sean Owen
> Fix For: 0.3
>
> Attachments: MAHOUT-191-patch.txt, MAHOUT-191.patch
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Sushil Bajracharya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sushil Bajracharya updated MAHOUT-191:
--------------------------------------
Attachment: MAHOUT-191-patch.txt
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Attachments: MAHOUT-191-patch.txt
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-191:
-----------------------------
Fix Version/s: 0.3
Assignee: Sean Owen
So, may I submit these on y'all's behalf? Not clear on the status here.
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Assignee: Sean Owen
> Fix For: 0.3
>
> Attachments: MAHOUT-191-patch.txt, MAHOUT-191.patch
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-191:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed variant on patches on behalf of submitters. Did not override the standard setDocumentNumber() method since this seems to be available in Lucene now and undesirable to cut off calls to super.setDocumentNumber()?
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Assignee: Sean Owen
> Fix For: 0.3
>
> Attachments: MAHOUT-191-patch.txt, MAHOUT-191.patch
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Sushil Bajracharya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sushil Bajracharya updated MAHOUT-191:
--------------------------------------
Status: Patch Available (was: Open)
It seems that the problem is because that not all the documents in my index has the field that I am using to get term vectors from. I made the following changes to make this work, but I am not sure if thats the right way. I wanted to get this work to run the LDA topic modeling using the output from the Driver.
Index: utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java
===================================================================
--- utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java (revision 830343)
+++ utils/src/main/java/org/apache/mahout/utils/vectors/io/SequenceFileVectorWriter.java (working copy)
@@ -42,7 +42,7 @@
break;
}
//point.write(dataOut);
- writer.append(new LongWritable(recNum++), point);
+ if(point!=null) writer.append(new LongWritable(recNum++), point);
}
return recNum;
Index: utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java
===================================================================
--- utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java (revision 830343)
+++ utils/src/main/java/org/apache/mahout/utils/vectors/lucene/LuceneIterable.java (working copy)
@@ -104,6 +104,10 @@
try {
indexReader.getTermFreqVector(doc, field, mapper);
result = mapper.getVector();
+
+ if (result == null)
+ return null;
+
if (idField != null) {
String id = indexReader.document(doc, idFieldSelector).get(idField);
result.setName(id);
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-191) NPE while creating term vectors with
an index on a field that does not exist in all the documents
Posted by "Shashikant Kore (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771408#action_12771408 ]
Shashikant Kore commented on MAHOUT-191:
----------------------------------------
I noticed a different problem for empty field. TFDFMapper returned vector for previous non-empty document. This happens because setExpectations() is not getting called on empty documents.
I have attached a patch to addres the same. Couldn't find a better way. If you know, please update accordingly.
> NPE while creating term vectors with an index on a field that does not exist in all the documents
> -------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-191
> URL: https://issues.apache.org/jira/browse/MAHOUT-191
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: mac, snow leopard, eclipse galileo, jdk 6
> Reporter: Sushil Bajracharya
> Attachments: MAHOUT-191-patch.txt, MAHOUT-191.patch
>
>
> (based on the message from here: http://www.nabble.com/Creating-Vectors-from-Text-tt24298643.html#a26090263)
> I checked out mahout from trunk and tried to create term frequency vector from a lucene index and ran into this..
> 09/10/27 17:36:10 INFO lucene.Driver: Output File: /Users/shoeseal/DATA/luc2tvec.out
> 09/10/27 17:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 09/10/27 17:36:11 INFO compress.CodecPool: Got brand-new compressor
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:109)
> at org.apache.mahout.utils.vectors.lucene.LuceneIterable$TDIterator.next(LuceneIterable.java:1)
> at org.apache.mahout.utils.vectors.io.SequenceFileVectorWriter.write(SequenceFileVectorWriter.java:40)
> at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:200)
> I am running this from Eclipse (snow leopard with JDK 6), on an index that has field with stored term vectors..
> my input parameters for Driver are:
> --dir <path>/smallidx/ --output <path>/luc2tvec.out --idField id_field
> --field field_with_TV --dictOut <path>/luc2tvec.dict --max 50 --weight tf
> Luke shows the following info on the fields I am using:
> id_field is indexed, stored, omit norms
> field_with_TV is indexed, tokenized, stored, term vector
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.