You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-dev@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2008/12/07 01:23:08 UTC

Generating Token factories

I was trying to run the Analyzer stub generation at http://wiki.apache.org/solr/CommitterInfo#head-c2ba467b48dcfd17c59d09a2ad70f4c4fffb4ce8

and it resulted in:
stub-factories:
      [exec] /Volumes/User/grantingersoll/projects/lucene/solr/solr- 
clean/src/java/org/apache/solr/analysis/CharStreamAwareCJKTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/solr/solr- 
clean/src/java/org/apache/solr/analysis/ 
CharStreamAwareWhitespaceTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/ar/ 
ArabicLetterTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/ar/ 
ArabicNormalizationFilter.java
      [exec] can't stub ArabicNormalizationFilter
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/ar/ 
ArabicStemFilter.java
      [exec] can't stub ArabicStemFilter
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/sinks/ 
DateRecognizerSinkTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/sinks/ 
TokenRangeSinkTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/contrib/analyzers/src/java/org/apache/lucene/analysis/sinks/ 
TokenTypeSinkTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/src/java/org/apache/lucene/analysis/SinkTokenizer.java
      [exec] /Volumes/User/grantingersoll/projects/lucene/java/lucene- 
clean/src/java/org/apache/lucene/analysis/TeeTokenFilter.java
      [exec] Can't find java files for...
      [exec] org.apache.lucene.analysis.sinks.TokenRangeSinkTokenizer
      [exec] org.apache.lucene.analysis.SinkTokenizer
      [exec] org.apache.lucene.analysis.ar.ArabicNormalizationFilter
      [exec] org.apache.lucene.analysis.ar.ArabicStemFilter
      [exec] org.apache.solr.analysis.CharStreamAwareWhitespaceTokenizer
      [exec] org.apache.solr.analysis.CharStreamAwareCJKTokenizer
      [exec] org.apache.lucene.analysis.sinks.TokenTypeSinkTokenizer
      [exec]  
org.apache.lucene.analysis.sinks.DateRecognizerSinkTokenizer


I don't see much in the way of reasons why, so perhaps Hoss can lend  
some insight.

I'm mostly interested in generating the Arabic ones, but I guess I  
will do them by hand.

Thanks,
Grant

Re: Generating Token factories

Posted by Grant Ingersoll <gs...@apache.org>.

On Dec 9, 2008, at 2:00 AM, Chris Hostetter wrote:

>
> :      [exec] can't stub ArabicNormalizationFilter
> 	...
> :      [exec] can't stub ArabicStemFilter
>
> ...those messages are because those Filter's don't have public
> constructors ... i must be really tired because i can't for the life  
> of me
> understand how the manually made Factories you checked in are  
> working in
> spite of this.

I updated them to be public and checked in a new Lucene jar.  You  
probably didn't do a full svn up.

>
>
> :      [exec] Can't find java files for...
>
> some of that is because the CharStreamAware*Factories we already had
> didn't declare a specific return type, and some of that was because i
> aparently botched the ignore regex for the Sink and Tee fitlers
>
> I've committed a few error message fixes to the script (r724624) and  
> made
> all the factories have specific return types (r724625) as well as  
> adding
> the ASCII Filter (r724626)
>

Thanks.

>
> (I really don't understand those Arabic Filters without public
> constructors though.)
>

It's all magic ;-)  Nothing to see here, please move along!


-Grant

Re: Generating Token factories

Posted by Chris Hostetter <ho...@fucit.org>.

:      [exec] can't stub ArabicNormalizationFilter
	...
:      [exec] can't stub ArabicStemFilter

...those messages are because those Filter's don't have public 
constructors ... i must be really tired because i can't for the life of me 
understand how the manually made Factories you checked in are working in 
spite of this.

:      [exec] Can't find java files for...

some of that is because the CharStreamAware*Factories we already had 
didn't declare a specific return type, and some of that was because i  
aparently botched the ignore regex for the Sink and Tee fitlers

I've committed a few error message fixes to the script (r724624) and made 
all the factories have specific return types (r724625) as well as adding 
the ASCII Filter (r724626)


(I really don't understand those Arabic Filters without public 
constructors though.)


-Hoss