You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Christopher Jordan <cj...@gilt.com> on 2011/04/18 19:52:26 UTC
LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Hi,
I opened a rather detailed JIRA ticket and submitted patch regarding this issue already:
https://issues.apache.org/jira/browse/MAHOUT-675
The short of it is that the LuceneIterator throws an IllegalStateException when a null term vector is encountered in the computeNext method. That is problematic as it pushes the responsibility of checking if a document field has no legitimate terms onto the person creating and maintaining the Lucene index. That might sound good in principle however, it isn't exactly intuitive to do that when creating or maintaining a Lucene index. The reason this is even an issue is that when you start creating custom Lucene analyzers, a pretty important practice if you want to improve your text mining results, it is possible that you will end up filtering out all terms in the target field for some documents; that is actually a desirable result as it indicates that those document is noise. Thus, when you attempt to dump the vectors of that index, the noise documents cause an IllegalStateException and it does not indicate that the issue was due the the custom analyzer.
I believe, at least in my situation, a better approach is for the LuceneIterator to log a warning with the idField when it encounters a problem document and move onto the next one.
Thanks,
Chris
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
I have some code half done to do this. SHould post a patch this morning.
On Tue, Apr 19, 2011 at 8:40 AM, Ted Dunning <te...@gmail.com> wrote:
> Btw... my only suggestion on this particular patch is that it would be a
> bit better to have the log messages on failure back off in frequency so that
> the total number of log messages isn't proportional to the number of errors.
> I tend to do power of ten or 1,2,5 back-off for these things.
>
>
> On Tue, Apr 19, 2011 at 6:25 AM, Christopher Jordan <cj...@gilt.com>wrote:
>
>> Just to further the point, logging is quite important. While you obviously
>> will not review every log, in a production environment, you certainly will
>> have monitoring scripts check them for ERROR and WARN entries. As well, if
>> you do not want to see the WARN entries from a specific class, you can
>> configure your logger to skip over them.
>>
>> On Apr 19, 2011, at 12:07 AM, Ted Dunning wrote:
>>
>> > I disagree. You should document that you are discarding documents. It
>> is
>> > reasonable to not document every lost document and good to throw an
>> > exception when too many failures occur.
>> >
>> > It is almost inevitable with large data that some inputs are malformed.
>> > These can't stop the show, but you have to know what your exception rate
>> is
>> > so you can detect catastrophic failures.
>> >
>> > On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com>
>> wrote:
>> >
>> >> Please don't log it. Nobody reads logs.
>> >> Right is right and wrong is wrong. Either throw an exception or ignore
>> it.
>> >> You can include a ratio of accepted vectors as an output.
>> >>
>> >> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>
>> >> wrote:
>> >>> I have incorporated this requested change in a new patch that I
>> attached
>> >> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
>> >>>
>> >>> It appears that the previous patch has already been applied. Should I
>> >> repull the repo, make a new ticket, and create a new patch?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Chris
>> >>>
>> >>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
>> >>>
>> >>> That sounds right to me.
>> >>>
>> >>> It might be plausible to blow an exception if a (configurable) large
>> >> percentage of all documents have to be rejected. That is a minor
>> >> improvement, though.
>> >>>
>> >>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <
>> cjordan@gilt.com
>> >> <ma...@gilt.com>> wrote:
>> >>> I believe, at least in my situation, a better approach is for the
>> >> LuceneIterator to log a warning with the idField when it encounters a
>> >> problem document and move onto the next one.
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Lance Norskog
>> >> goksron@gmail.com
>> >>
>>
>>
>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
Btw... my only suggestion on this particular patch is that it would be a bit
better to have the log messages on failure back off in frequency so that the
total number of log messages isn't proportional to the number of errors. I
tend to do power of ten or 1,2,5 back-off for these things.
On Tue, Apr 19, 2011 at 6:25 AM, Christopher Jordan <cj...@gilt.com>wrote:
> Just to further the point, logging is quite important. While you obviously
> will not review every log, in a production environment, you certainly will
> have monitoring scripts check them for ERROR and WARN entries. As well, if
> you do not want to see the WARN entries from a specific class, you can
> configure your logger to skip over them.
>
> On Apr 19, 2011, at 12:07 AM, Ted Dunning wrote:
>
> > I disagree. You should document that you are discarding documents. It
> is
> > reasonable to not document every lost document and good to throw an
> > exception when too many failures occur.
> >
> > It is almost inevitable with large data that some inputs are malformed.
> > These can't stop the show, but you have to know what your exception rate
> is
> > so you can detect catastrophic failures.
> >
> > On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com>
> wrote:
> >
> >> Please don't log it. Nobody reads logs.
> >> Right is right and wrong is wrong. Either throw an exception or ignore
> it.
> >> You can include a ratio of accepted vectors as an output.
> >>
> >> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>
> >> wrote:
> >>> I have incorporated this requested change in a new patch that I
> attached
> >> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
> >>>
> >>> It appears that the previous patch has already been applied. Should I
> >> repull the repo, make a new ticket, and create a new patch?
> >>>
> >>> Thanks,
> >>>
> >>> Chris
> >>>
> >>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
> >>>
> >>> That sounds right to me.
> >>>
> >>> It might be plausible to blow an exception if a (configurable) large
> >> percentage of all documents have to be rejected. That is a minor
> >> improvement, though.
> >>>
> >>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cjordan@gilt.com
> >> <ma...@gilt.com>> wrote:
> >>> I believe, at least in my situation, a better approach is for the
> >> LuceneIterator to log a warning with the idField when it encounters a
> >> problem document and move onto the next one.
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Lance Norskog
> >> goksron@gmail.com
> >>
>
>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Christopher Jordan <cj...@gilt.com>.
Thanks Ted,
I found a minor bug with the logging logic. I posted a revised patch to the ticket.
I think the next step here is to update the LuceneIterable and the Driver to allow for configuration of these new attributes. I think I will wait for the patch in MAHOUT-671 to get committed first though.
Chris
On Apr 19, 2011, at 4:55 PM, Ted Dunning wrote:
Chris,
Can you review the patch I just pushed up that adjusts how much logging is produced.
On Tue, Apr 19, 2011 at 6:25 AM, Christopher Jordan <cj...@gilt.com>> wrote:
Just to further the point, logging is quite important. While you obviously will not review every log, in a production environment, you certainly will have monitoring scripts check them for ERROR and WARN entries. As well, if you do not want to see the WARN entries from a specific class, you can configure your logger to skip over them.
On Apr 19, 2011, at 12:07 AM, Ted Dunning wrote:
> I disagree. You should document that you are discarding documents. It is
> reasonable to not document every lost document and good to throw an
> exception when too many failures occur.
>
> It is almost inevitable with large data that some inputs are malformed.
> These can't stop the show, but you have to know what your exception rate is
> so you can detect catastrophic failures.
>
> On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com>> wrote:
>
>> Please don't log it. Nobody reads logs.
>> Right is right and wrong is wrong. Either throw an exception or ignore it.
>> You can include a ratio of accepted vectors as an output.
>>
>> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>>
>> wrote:
>>> I have incorporated this requested change in a new patch that I attached
>> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
>>>
>>> It appears that the previous patch has already been applied. Should I
>> repull the repo, make a new ticket, and create a new patch?
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
>>>
>>> That sounds right to me.
>>>
>>> It might be plausible to blow an exception if a (configurable) large
>> percentage of all documents have to be rejected. That is a minor
>> improvement, though.
>>>
>>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>
>> <ma...@gilt.com>>> wrote:
>>> I believe, at least in my situation, a better approach is for the
>> LuceneIterator to log a warning with the idField when it encounters a
>> problem document and move onto the next one.
>>>
>>>
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com<ma...@gmail.com>
>>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
Chris,
Can you review the patch I just pushed up that adjusts how much logging is
produced.
On Tue, Apr 19, 2011 at 6:25 AM, Christopher Jordan <cj...@gilt.com>wrote:
> Just to further the point, logging is quite important. While you obviously
> will not review every log, in a production environment, you certainly will
> have monitoring scripts check them for ERROR and WARN entries. As well, if
> you do not want to see the WARN entries from a specific class, you can
> configure your logger to skip over them.
>
> On Apr 19, 2011, at 12:07 AM, Ted Dunning wrote:
>
> > I disagree. You should document that you are discarding documents. It
> is
> > reasonable to not document every lost document and good to throw an
> > exception when too many failures occur.
> >
> > It is almost inevitable with large data that some inputs are malformed.
> > These can't stop the show, but you have to know what your exception rate
> is
> > so you can detect catastrophic failures.
> >
> > On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com>
> wrote:
> >
> >> Please don't log it. Nobody reads logs.
> >> Right is right and wrong is wrong. Either throw an exception or ignore
> it.
> >> You can include a ratio of accepted vectors as an output.
> >>
> >> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>
> >> wrote:
> >>> I have incorporated this requested change in a new patch that I
> attached
> >> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
> >>>
> >>> It appears that the previous patch has already been applied. Should I
> >> repull the repo, make a new ticket, and create a new patch?
> >>>
> >>> Thanks,
> >>>
> >>> Chris
> >>>
> >>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
> >>>
> >>> That sounds right to me.
> >>>
> >>> It might be plausible to blow an exception if a (configurable) large
> >> percentage of all documents have to be rejected. That is a minor
> >> improvement, though.
> >>>
> >>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cjordan@gilt.com
> >> <ma...@gilt.com>> wrote:
> >>> I believe, at least in my situation, a better approach is for the
> >> LuceneIterator to log a warning with the idField when it encounters a
> >> problem document and move onto the next one.
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Lance Norskog
> >> goksron@gmail.com
> >>
>
>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Christopher Jordan <cj...@gilt.com>.
Just to further the point, logging is quite important. While you obviously will not review every log, in a production environment, you certainly will have monitoring scripts check them for ERROR and WARN entries. As well, if you do not want to see the WARN entries from a specific class, you can configure your logger to skip over them.
On Apr 19, 2011, at 12:07 AM, Ted Dunning wrote:
> I disagree. You should document that you are discarding documents. It is
> reasonable to not document every lost document and good to throw an
> exception when too many failures occur.
>
> It is almost inevitable with large data that some inputs are malformed.
> These can't stop the show, but you have to know what your exception rate is
> so you can detect catastrophic failures.
>
> On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com> wrote:
>
>> Please don't log it. Nobody reads logs.
>> Right is right and wrong is wrong. Either throw an exception or ignore it.
>> You can include a ratio of accepted vectors as an output.
>>
>> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>
>> wrote:
>>> I have incorporated this requested change in a new patch that I attached
>> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
>>>
>>> It appears that the previous patch has already been applied. Should I
>> repull the repo, make a new ticket, and create a new patch?
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
>>>
>>> That sounds right to me.
>>>
>>> It might be plausible to blow an exception if a (configurable) large
>> percentage of all documents have to be rejected. That is a minor
>> improvement, though.
>>>
>>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cjordan@gilt.com
>> <ma...@gilt.com>> wrote:
>>> I believe, at least in my situation, a better approach is for the
>> LuceneIterator to log a warning with the idField when it encounters a
>> problem document and move onto the next one.
>>>
>>>
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
I disagree. You should document that you are discarding documents. It is
reasonable to not document every lost document and good to throw an
exception when too many failures occur.
It is almost inevitable with large data that some inputs are malformed.
These can't stop the show, but you have to know what your exception rate is
so you can detect catastrophic failures.
On Mon, Apr 18, 2011 at 6:00 PM, Lance Norskog <go...@gmail.com> wrote:
> Please don't log it. Nobody reads logs.
> Right is right and wrong is wrong. Either throw an exception or ignore it.
> You can include a ratio of accepted vectors as an output.
>
> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>
> wrote:
> > I have incorporated this requested change in a new patch that I attached
> to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
> >
> > It appears that the previous patch has already been applied. Should I
> repull the repo, make a new ticket, and create a new patch?
> >
> > Thanks,
> >
> > Chris
> >
> > On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
> >
> > That sounds right to me.
> >
> > It might be plausible to blow an exception if a (configurable) large
> percentage of all documents have to be rejected. That is a minor
> improvement, though.
> >
> > On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cjordan@gilt.com
> <ma...@gilt.com>> wrote:
> > I believe, at least in my situation, a better approach is for the
> LuceneIterator to log a warning with the idField when it encounters a
> problem document and move onto the next one.
> >
> >
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: LuceneIterator throws an IllegalStateException when a null term frequency vector is encountered
Posted by Grant Ingersoll <gs...@apache.org>.
I'd rather a person had to explicitly set the behavior to ignore the exception, as in most situations it would be better to get the exception, although I agree w/ Christopher's arguments that in some cases you just want to go along.
On Apr 19, 2011, at 3:00 AM, Lance Norskog wrote:
> Please don't log it. Nobody reads logs.
> Right is right and wrong is wrong. Either throw an exception or ignore it.
> You can include a ratio of accepted vectors as an output.
>
> On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com> wrote:
>> I have incorporated this requested change in a new patch that I attached to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
>>
>> It appears that the previous patch has already been applied. Should I repull the repo, make a new ticket, and create a new patch?
>>
>> Thanks,
>>
>> Chris
>>
>> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
>>
>> That sounds right to me.
>>
>> It might be plausible to blow an exception if a (configurable) large percentage of all documents have to be rejected. That is a minor improvement, though.
>>
>> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>> wrote:
>> I believe, at least in my situation, a better approach is for the LuceneIterator to log a warning with the idField when it encounters a problem document and move onto the next one.
>>
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Lance Norskog <go...@gmail.com>.
Please don't log it. Nobody reads logs.
Right is right and wrong is wrong. Either throw an exception or ignore it.
You can include a ratio of accepted vectors as an output.
On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com> wrote:
> I have incorporated this requested change in a new patch that I attached to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
>
> It appears that the previous patch has already been applied. Should I repull the repo, make a new ticket, and create a new patch?
>
> Thanks,
>
> Chris
>
> On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
>
> That sounds right to me.
>
> It might be plausible to blow an exception if a (configurable) large percentage of all documents have to be rejected. That is a minor improvement, though.
>
> On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>> wrote:
> I believe, at least in my situation, a better approach is for the LuceneIterator to log a warning with the idField when it encounters a problem document and move onto the next one.
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
Yeah... that sounds right.
On Mon, Apr 18, 2011 at 5:52 PM, Christopher Jordan <cj...@gilt.com>wrote:
> It appears that the previous patch has already been applied. Should I
> repull the repo, make a new ticket, and create a new patch?
>
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Christopher Jordan <cj...@gilt.com>.
I have incorporated this requested change in a new patch that I attached to ticket https://issues.apache.org/jira/browse/MAHOUT-675.
It appears that the previous patch has already been applied. Should I repull the repo, make a new ticket, and create a new patch?
Thanks,
Chris
On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
That sounds right to me.
It might be plausible to blow an exception if a (configurable) large percentage of all documents have to be rejected. That is a minor improvement, though.
On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>> wrote:
I believe, at least in my situation, a better approach is for the LuceneIterator to log a warning with the idField when it encounters a problem document and move onto the next one.
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Christopher Jordan <cj...@gilt.com>.
Sounds like a good idea with the default set to 100%. I'll make that update.
On Apr 18, 2011, at 1:54 PM, Ted Dunning wrote:
That sounds right to me.
It might be plausible to blow an exception if a (configurable) large percentage of all documents have to be rejected. That is a minor improvement, though.
On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>> wrote:
I believe, at least in my situation, a better approach is for the LuceneIterator to log a warning with the idField when it encounters a problem document and move onto the next one.
Re: LuceneIterator throws an IllegalStateException when a null term
frequency vector is encountered
Posted by Ted Dunning <te...@gmail.com>.
That sounds right to me.
It might be plausible to blow an exception if a (configurable) large
percentage of all documents have to be rejected. That is a minor
improvement, though.
On Mon, Apr 18, 2011 at 10:52 AM, Christopher Jordan <cj...@gilt.com>wrote:
> I believe, at least in my situation, a better approach is for the
> LuceneIterator to log a warning with the idField when it encounters a
> problem document and move onto the next one.
>