You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Ryan McKinley <ry...@gmail.com> on 2009/08/21 16:13:24 UTC

/trunk ClassCastException with BaseTokenizerFactory

Just updated to /trunk and am now seeing this exception:

Caused by: org.apache.solr.client.solrj.SolrServerException:
java.lang.ClassCastException:
xxx.solr.analysis.JSONKeyValueTokenizerFactory$1 cannot be cast to
org.apache.lucene.analysis.Tokenizer
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
	... 15 more
Caused by: java.lang.ClassCastException:
xxx.solr.analysis.JSONKeyValueTokenizerFactory$1 cannot be cast to
org.apache.lucene.analysis.Tokenizer
	at org.apache.solr.analysis.TokenizerChain.getStream(TokenizerChain.java:69)
	at org.apache.solr.analysis.SolrAnalyzer.reusableTokenStream(SolrAnalyzer.java:74)
	at org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer.reusableTokenStream(IndexSchema.java:364)
	at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:124)
	at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:244)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)


Looks like SolrIndexAnalyzer now assumes everything uses the new
TokenStream API...

I'm fine upgrading, but it seems we should the 'back compatibility'
notice more explicit.


FYI, this is what the TokenizerFactory looks like:

public class JSONKeyValueTokenizerFactory extends BaseTokenizerFactory
{
  ...

  public TokenStream create(Reader input) {
    final JSONParser js = new JSONParser( input );
    final Stack<String> keystack = new Stack<String>();

    return new TokenStream()
    {
      ...

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Ryan McKinley <ry...@gmail.com>.
On Aug 21, 2009, at 10:49 AM, Yonik Seeley wrote:

> On Fri, Aug 21, 2009 at 10:33 AM, Ryan McKinley<ry...@gmail.com>  
> wrote:
>> Actually.... I think there may be something wrong here.
>>
>> BaseTokenizerFactory does not make a Tokenizer, it creates a
>> TokenStream, so it should never be cast to Tokenizer
>>
>> My custom TokenizerFactory now looks the same as:
>> o.a.s.analysis.PatternTokenizerFactory
>
> Urg... looks like there's no end-to-end (index then search) test for
> PatternTokenizerFactory, so we never caught this.

I guess we need to add one :)

>
> It seems like when something is specified as a <tokenizer> in
> schema.xml it should in fact be a tokenizer - it's the only way
> tokenstream reuse works.
>

I don't see anything in Solr that creates a Tokenizer.  The  
TokenizerFactory just creates a TokenStream.

It seems that TokenizerFactory really needs to be:
   public Tokenizer create( Reader input )
rather then:
   public TokenStream create( Reader input );

I don't see any backwards compatible way to make this change!

ideas?
ryan

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Yonik Seeley <ys...@gmail.com>.
On Fri, Aug 21, 2009 at 10:33 AM, Ryan McKinley<ry...@gmail.com> wrote:
> Actually.... I think there may be something wrong here.
>
> BaseTokenizerFactory does not make a Tokenizer, it creates a
> TokenStream, so it should never be cast to Tokenizer
>
> My custom TokenizerFactory now looks the same as:
> o.a.s.analysis.PatternTokenizerFactory

Urg... looks like there's no end-to-end (index then search) test for
PatternTokenizerFactory, so we never caught this.

It seems like when something is specified as a <tokenizer> in
schema.xml it should in fact be a tokenizer - it's the only way
tokenstream reuse works.

-Yonik

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Ryan McKinley <ry...@gmail.com>.
Actually.... I think there may be something wrong here.

BaseTokenizerFactory does not make a Tokenizer, it creates a
TokenStream, so it should never be cast to Tokenizer

My custom TokenizerFactory now looks the same as:
o.a.s.analysis.PatternTokenizerFactory

Not sure what to look at next...  ideas?

thanks
ryan


On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com> wrote:
> Just updated to /trunk and am now seeing this exception:
>
> Caused by: org.apache.solr.client.solrj.SolrServerException:
> java.lang.ClassCastException:
> xxx.solr.analysis.JSONKeyValueTokenizerFactory$1 cannot be cast to
> org.apache.lucene.analysis.Tokenizer
>        at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:141)
>        ... 15 more
> Caused by: java.lang.ClassCastException:
> xxx.solr.analysis.JSONKeyValueTokenizerFactory$1 cannot be cast to
> org.apache.lucene.analysis.Tokenizer
>        at org.apache.solr.analysis.TokenizerChain.getStream(TokenizerChain.java:69)
>        at org.apache.solr.analysis.SolrAnalyzer.reusableTokenStream(SolrAnalyzer.java:74)
>        at org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer.reusableTokenStream(IndexSchema.java:364)
>        at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:124)
>        at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:244)
>        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
>
>
> Looks like SolrIndexAnalyzer now assumes everything uses the new
> TokenStream API...
>
> I'm fine upgrading, but it seems we should the 'back compatibility'
> notice more explicit.
>
>
> FYI, this is what the TokenizerFactory looks like:
>
> public class JSONKeyValueTokenizerFactory extends BaseTokenizerFactory
> {
>  ...
>
>  public TokenStream create(Reader input) {
>    final JSONParser js = new JSONParser( input );
>    final Stack<String> keystack = new Stack<String>();
>
>    return new TokenStream()
>    {
>      ...
>

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Ryan McKinley <ry...@gmail.com>.
not sure what you are asking for.

The patch in SOLR-1377 (now /trunk) breaks the TokenizerFactory API.

You will get an compiler error (or runtime error) with a  
TokenizerFactory that creates a TokenStream rather then a Tokenizer.

I don't see any clean way to do this via deprecation -- if you have  
ideas, shout them out.

ryan



On Aug 24, 2009, at 1:49 PM, Lance Norskog wrote:

> Is it possible to mark out backwards-incompatible changes with  
> deprecation?
> So at least we get warnings in Eclipse/Netbeans etc?
>
> On Fri, Aug 21, 2009 at 9:50 AM, Mark Miller <ma...@gmail.com>  
> wrote:
>
>> Ryan McKinley wrote:
>>> Ahh, I see:
>>> Tokenizer extends TokenStream
>>>
>>> So if this is going to break everything that implements TokenStream
>>> rather then Tokenizer, it seems we should change the  
>>> TokenizerFactory
>>> API from:
>>>  public Tokenizer create( Reader input )
>>> rather then:
>>>  public TokenStream create( Reader input );
>>>
>>> I would WAY rather have my compiler tell me something is wrong then
>>> get an error and then find some documentation about the tokenizer.
>>>
>>> - - - - -
>>>
>>> Personally, I think lucene/solr just need to fess up and admit that
>>> 2.9 is *not* totally back compatible.
>> I don't think anyone contends that Lucene is totally backcompat - and
>> insofarasthatgoes there is no way Solr totally is - . it exposes a  
>> lot
>> of Lucene.
>>
>> We admit our breaks in this release in the back compat breaks  
>> section.
>> There is no way we will release claiming total back compat. Not  
>> even in
>> the realm of possibility.
>>> No way is the Multireader change back-compatible!
>>
>> Personally, pure API wise - I think it was. Its a stickier issue on  
>> the
>> possible more RAM usage - but too me, thats more of a Runtime change.
>> Certain methods have always changed over time in their resource  
>> usage,
>> and I think thats within back compat. This was a steep one to swallow
>> though, I'll admit. Basically we just thought it was way worth it  
>> long
>> term. And Hoss came up with some great ideas to help ease the  
>> possible
>> pain.
>>>
>>> ryan
>>>
>>>
>>> On Aug 21, 2009, at 11:39 AM, Yonik Seeley wrote:
>>>
>>>> On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com>
>>>> wrote:
>>>>> I'm fine upgrading, but it seems we should the 'back  
>>>>> compatibility'
>>>>> notice more explicit.
>>>>
>>>> Yeah... that should be fun for expert-use plugins in general.  In
>>>> Lucene-land, this is the release of the "break"... I think we've
>>>> covered the changes reasonably well in our external APIs, but  
>>>> people
>>>> can always use pretty much the full Lucene API when writing Solr
>>>> plugins.
>>>>
>>>> I think we'll need to document that things in <tokenizer> tags  
>>>> need to
>>>> inherit from Tokenizer classes.  It is technically a back-compat
>>>> break, but I assume it will affect very few users?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>
>
> -- 
> Lance Norskog
> goksron@gmail.com


Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Lance Norskog <go...@gmail.com>.
Is it possible to mark out backwards-incompatible changes with deprecation?
So at least we get warnings in Eclipse/Netbeans etc?

On Fri, Aug 21, 2009 at 9:50 AM, Mark Miller <ma...@gmail.com> wrote:

> Ryan McKinley wrote:
> > Ahh, I see:
> >  Tokenizer extends TokenStream
> >
> > So if this is going to break everything that implements TokenStream
> > rather then Tokenizer, it seems we should change the TokenizerFactory
> > API from:
> >   public Tokenizer create( Reader input )
> > rather then:
> >   public TokenStream create( Reader input );
> >
> > I would WAY rather have my compiler tell me something is wrong then
> > get an error and then find some documentation about the tokenizer.
> >
> > - - - - -
> >
> > Personally, I think lucene/solr just need to fess up and admit that
> > 2.9 is *not* totally back compatible.
> I don't think anyone contends that Lucene is totally backcompat - and
> insofarasthatgoes there is no way Solr totally is - . it exposes a lot
> of Lucene.
>
> We admit our breaks in this release in the back compat breaks section.
> There is no way we will release claiming total back compat. Not even in
> the realm of possibility.
> > No way is the Multireader change back-compatible!
>
> Personally, pure API wise - I think it was. Its a stickier issue on the
> possible more RAM usage - but too me, thats more of a Runtime change.
> Certain methods have always changed over time in their resource usage,
> and I think thats within back compat. This was a steep one to swallow
> though, I'll admit. Basically we just thought it was way worth it long
> term. And Hoss came up with some great ideas to help ease the possible
> pain.
> >
> > ryan
> >
> >
> > On Aug 21, 2009, at 11:39 AM, Yonik Seeley wrote:
> >
> >> On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com>
> >> wrote:
> >>> I'm fine upgrading, but it seems we should the 'back compatibility'
> >>> notice more explicit.
> >>
> >> Yeah... that should be fun for expert-use plugins in general.  In
> >> Lucene-land, this is the release of the "break"... I think we've
> >> covered the changes reasonably well in our external APIs, but people
> >> can always use pretty much the full Lucene API when writing Solr
> >> plugins.
> >>
> >> I think we'll need to document that things in <tokenizer> tags need to
> >> inherit from Tokenizer classes.  It is technically a back-compat
> >> break, but I assume it will affect very few users?
> >>
> >> -Yonik
> >> http://www.lucidimagination.com
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>


-- 
Lance Norskog
goksron@gmail.com

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Mark Miller <ma...@gmail.com>.
Ryan McKinley wrote:
> Ahh, I see:
>  Tokenizer extends TokenStream
>
> So if this is going to break everything that implements TokenStream
> rather then Tokenizer, it seems we should change the TokenizerFactory
> API from:
>   public Tokenizer create( Reader input )
> rather then:
>   public TokenStream create( Reader input );
>
> I would WAY rather have my compiler tell me something is wrong then
> get an error and then find some documentation about the tokenizer.
>
> - - - - -
>
> Personally, I think lucene/solr just need to fess up and admit that
> 2.9 is *not* totally back compatible.  
I don't think anyone contends that Lucene is totally backcompat - and
insofarasthatgoes there is no way Solr totally is - . it exposes a lot
of Lucene.

We admit our breaks in this release in the back compat breaks section.
There is no way we will release claiming total back compat. Not even in
the realm of possibility.
> No way is the Multireader change back-compatible!

Personally, pure API wise - I think it was. Its a stickier issue on the
possible more RAM usage - but too me, thats more of a Runtime change.
Certain methods have always changed over time in their resource usage,
and I think thats within back compat. This was a steep one to swallow
though, I'll admit. Basically we just thought it was way worth it long
term. And Hoss came up with some great ideas to help ease the possible pain.
>
> ryan
>
>
> On Aug 21, 2009, at 11:39 AM, Yonik Seeley wrote:
>
>> On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com>
>> wrote:
>>> I'm fine upgrading, but it seems we should the 'back compatibility'
>>> notice more explicit.
>>
>> Yeah... that should be fun for expert-use plugins in general.  In
>> Lucene-land, this is the release of the "break"... I think we've
>> covered the changes reasonably well in our external APIs, but people
>> can always use pretty much the full Lucene API when writing Solr
>> plugins.
>>
>> I think we'll need to document that things in <tokenizer> tags need to
>> inherit from Tokenizer classes.  It is technically a back-compat
>> break, but I assume it will affect very few users?
>>
>> -Yonik
>> http://www.lucidimagination.com
>


-- 
- Mark

http://www.lucidimagination.com




Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Yonik Seeley <ys...@gmail.com>.
On Fri, Aug 21, 2009 at 12:22 PM, Ryan McKinley<ry...@gmail.com> wrote:
> Ahh, I see:
>  Tokenizer extends TokenStream
>
> So if this is going to break everything that implements TokenStream rather
> then Tokenizer, it seems we should change the TokenizerFactory API from:
>  public Tokenizer create( Reader input )
> rather then:
>  public TokenStream create( Reader input );
>
> I would WAY rather have my compiler tell me something is wrong then get an
> error and then find some documentation about the tokenizer.

+1
Absolutely.

-Yonik
http://www.lucidimagination.com

Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Ryan McKinley <ry...@gmail.com>.
Ahh, I see:
  Tokenizer extends TokenStream

So if this is going to break everything that implements TokenStream  
rather then Tokenizer, it seems we should change the TokenizerFactory  
API from:
   public Tokenizer create( Reader input )
rather then:
   public TokenStream create( Reader input );

I would WAY rather have my compiler tell me something is wrong then  
get an error and then find some documentation about the tokenizer.

- - - - -

Personally, I think lucene/solr just need to fess up and admit that  
2.9 is *not* totally back compatible.  No way is the Multireader  
change back-compatible!

ryan


On Aug 21, 2009, at 11:39 AM, Yonik Seeley wrote:

> On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com>  
> wrote:
>> I'm fine upgrading, but it seems we should the 'back compatibility'
>> notice more explicit.
>
> Yeah... that should be fun for expert-use plugins in general.  In
> Lucene-land, this is the release of the "break"... I think we've
> covered the changes reasonably well in our external APIs, but people
> can always use pretty much the full Lucene API when writing Solr
> plugins.
>
> I think we'll need to document that things in <tokenizer> tags need to
> inherit from Tokenizer classes.  It is technically a back-compat
> break, but I assume it will affect very few users?
>
> -Yonik
> http://www.lucidimagination.com


Re: /trunk ClassCastException with BaseTokenizerFactory

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Aug 21, 2009 at 10:13 AM, Ryan McKinley<ry...@gmail.com> wrote:
> I'm fine upgrading, but it seems we should the 'back compatibility'
> notice more explicit.

Yeah... that should be fun for expert-use plugins in general.  In
Lucene-land, this is the release of the "break"... I think we've
covered the changes reasonably well in our external APIs, but people
can always use pretty much the full Lucene API when writing Solr
plugins.

I think we'll need to document that things in <tokenizer> tags need to
inherit from Tokenizer classes.  It is technically a back-compat
break, but I assume it will affect very few users?

-Yonik
http://www.lucidimagination.com