You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by PrasannaKumar Chamarty <te...@gmail.com> on 2016/04/12 10:37:03 UTC

Custom indexing

Hi,

What is the best way (in terms of maintenance required with new lucene
releases) to allow splitting of words on "." and "_" for indexing ? Thank
you.

Re: Custom indexing

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.

Hi,

Please try letter tokenizer, it should cover your example.

Ahmet

On Monday, April 18, 2016 3:02 PM, PK C <te...@gmail.com> wrote:



Hi,

   Thank you very much for your quick responses.

Jack Krupansky,

The main use case is searching in file names. For example, lucene.txt,
lucene_new.txt, lucene_1_new.txt. If I use 'lucene', I need to get all 3
files. with 'new' I need to get last two files. Please note that Standard
analyzer/tokenizer of lucene 3.6 is not giving us the results with
tokenization of  "." and "_". Are you referring to later versions than 3.6 ?

Ahmet,

1. Not sure if LetterTokenizer helps with the above example of having
numbers and letters in file names.
2. WordDelimeterFilter does not seem to be lucene 3.6
3. MappingCharFilter  is what I am already using overriding initReader
method in my CustomAnalyzer (Source copied from StandardAnalyzer (final
class)). Is this a good way to make use of final class StandardAnalyzer
with some custom changes ? Or composition is better ?

Thank you again,
Best Regards


On Tue, Apr 12, 2016 at 8:45 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> The standard analyzer/tokenizer should do a decent job of splitting on dot,
> hyphen, and underscore, in addition to whitespace and other punctuation.
>
> Can you post some specific test cases you are concerned with? (You should
> always run some test cases.)
>
> -- Jack Krupansky
>
> On Tue, Apr 12, 2016 at 10:35 AM, Ahmet Arslan <io...@yahoo.com.invalid>
> wrote:
>
> > Hi Chamarty,
> >
> > Well, there are a lot of options here.
> >
> > 1) Use LetterTokenizer
> > 2) Use WordDelimeterFilter combined with WhiteSpaceTokenizer
> > 3) Use MappingCharFilter to replace those characters with spaces
> > .
> > .
> > .
> >
> > Ahmet
> >
> >
> > On Tuesday, April 12, 2016 3:58 PM, PrasannaKumar Chamarty <
> > tech.kumarpch@gmail.com> wrote:
> >
> >
> >
> > Hi,
> >
> > What is the best way (in terms of maintenance required with new lucene
> > releases) to allow splitting of words on "." and "_" for indexing ? Thank
> > you.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Custom indexing

Posted by Jack Krupansky <ja...@gmail.com>.
You failed to disclose up front that you are using such an old release of
Lucene. Lucene is now on 6.0. I'll defer to others if they wish to provide
support for such an old release.

-- Jack Krupansky

On Mon, Apr 18, 2016 at 8:01 AM, PK C <te...@gmail.com> wrote:

> Hi,
>
>    Thank you very much for your quick responses.
>
> Jack Krupansky,
>
> The main use case is searching in file names. For example, lucene.txt,
> lucene_new.txt, lucene_1_new.txt. If I use 'lucene', I need to get all 3
> files. with 'new' I need to get last two files. Please note that Standard
> analyzer/tokenizer of lucene 3.6 is not giving us the results with
> tokenization of  "." and "_". Are you referring to later versions than 3.6
> ?
>
> Ahmet,
>
> 1. Not sure if LetterTokenizer helps with the above example of having
> numbers and letters in file names.
> 2. WordDelimeterFilter does not seem to be lucene 3.6
> 3. MappingCharFilter  is what I am already using overriding initReader
> method in my CustomAnalyzer (Source copied from StandardAnalyzer (final
> class)). Is this a good way to make use of final class StandardAnalyzer
> with some custom changes ? Or composition is better ?
>
> Thank you again,
> Best Regards
>
> On Tue, Apr 12, 2016 at 8:45 PM, Jack Krupansky <ja...@gmail.com>
> wrote:
>
> > The standard analyzer/tokenizer should do a decent job of splitting on
> dot,
> > hyphen, and underscore, in addition to whitespace and other punctuation.
> >
> > Can you post some specific test cases you are concerned with? (You should
> > always run some test cases.)
> >
> > -- Jack Krupansky
> >
> > On Tue, Apr 12, 2016 at 10:35 AM, Ahmet Arslan <iorixxx@yahoo.com.invalid
> >
> > wrote:
> >
> > > Hi Chamarty,
> > >
> > > Well, there are a lot of options here.
> > >
> > > 1) Use LetterTokenizer
> > > 2) Use WordDelimeterFilter combined with WhiteSpaceTokenizer
> > > 3) Use MappingCharFilter to replace those characters with spaces
> > > .
> > > .
> > > .
> > >
> > > Ahmet
> > >
> > >
> > > On Tuesday, April 12, 2016 3:58 PM, PrasannaKumar Chamarty <
> > > tech.kumarpch@gmail.com> wrote:
> > >
> > >
> > >
> > > Hi,
> > >
> > > What is the best way (in terms of maintenance required with new lucene
> > > releases) to allow splitting of words on "." and "_" for indexing ?
> Thank
> > > you.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

RE: Custom indexing

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> The main use case is searching in file names. For example, lucene.txt,
> lucene_new.txt, lucene_1_new.txt. If I use 'lucene', I need to get all 3
> files. with 'new' I need to get last two files. Please note that Standard
> analyzer/tokenizer of lucene 3.6 is not giving us the results with
> tokenization of  "." and "_". Are you referring to later versions than 3.6 ?

Hi StandardTokenizer in 3.6 is the old, non Unicode-compliant tokenizer classic tokenizer. In Lucene 4+ it is called "ClassicTokenizer" because it is still used by some users, but newer code should use the new StandardTokenizer. From Lucene 4 on, StandardTokenizer implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

This one is not available in such old Lucene versions, sorry. Your only chance is LetterTokenizer or write your own.

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Custom indexing

Posted by PK C <te...@gmail.com>.
Hi,

   Thank you very much for your quick responses.

Jack Krupansky,

The main use case is searching in file names. For example, lucene.txt,
lucene_new.txt, lucene_1_new.txt. If I use 'lucene', I need to get all 3
files. with 'new' I need to get last two files. Please note that Standard
analyzer/tokenizer of lucene 3.6 is not giving us the results with
tokenization of  "." and "_". Are you referring to later versions than 3.6 ?

Ahmet,

1. Not sure if LetterTokenizer helps with the above example of having
numbers and letters in file names.
2. WordDelimeterFilter does not seem to be lucene 3.6
3. MappingCharFilter  is what I am already using overriding initReader
method in my CustomAnalyzer (Source copied from StandardAnalyzer (final
class)). Is this a good way to make use of final class StandardAnalyzer
with some custom changes ? Or composition is better ?

Thank you again,
Best Regards

On Tue, Apr 12, 2016 at 8:45 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> The standard analyzer/tokenizer should do a decent job of splitting on dot,
> hyphen, and underscore, in addition to whitespace and other punctuation.
>
> Can you post some specific test cases you are concerned with? (You should
> always run some test cases.)
>
> -- Jack Krupansky
>
> On Tue, Apr 12, 2016 at 10:35 AM, Ahmet Arslan <io...@yahoo.com.invalid>
> wrote:
>
> > Hi Chamarty,
> >
> > Well, there are a lot of options here.
> >
> > 1) Use LetterTokenizer
> > 2) Use WordDelimeterFilter combined with WhiteSpaceTokenizer
> > 3) Use MappingCharFilter to replace those characters with spaces
> > .
> > .
> > .
> >
> > Ahmet
> >
> >
> > On Tuesday, April 12, 2016 3:58 PM, PrasannaKumar Chamarty <
> > tech.kumarpch@gmail.com> wrote:
> >
> >
> >
> > Hi,
> >
> > What is the best way (in terms of maintenance required with new lucene
> > releases) to allow splitting of words on "." and "_" for indexing ? Thank
> > you.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Re: Custom indexing

Posted by Jack Krupansky <ja...@gmail.com>.
The standard analyzer/tokenizer should do a decent job of splitting on dot,
hyphen, and underscore, in addition to whitespace and other punctuation.

Can you post some specific test cases you are concerned with? (You should
always run some test cases.)

-- Jack Krupansky

On Tue, Apr 12, 2016 at 10:35 AM, Ahmet Arslan <io...@yahoo.com.invalid>
wrote:

> Hi Chamarty,
>
> Well, there are a lot of options here.
>
> 1) Use LetterTokenizer
> 2) Use WordDelimeterFilter combined with WhiteSpaceTokenizer
> 3) Use MappingCharFilter to replace those characters with spaces
> .
> .
> .
>
> Ahmet
>
>
> On Tuesday, April 12, 2016 3:58 PM, PrasannaKumar Chamarty <
> tech.kumarpch@gmail.com> wrote:
>
>
>
> Hi,
>
> What is the best way (in terms of maintenance required with new lucene
> releases) to allow splitting of words on "." and "_" for indexing ? Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Custom indexing

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi Chamarty,

Well, there are a lot of options here.

1) Use LetterTokenizer
2) Use WordDelimeterFilter combined with WhiteSpaceTokenizer
3) Use MappingCharFilter to replace those characters with spaces
.
.
.

Ahmet


On Tuesday, April 12, 2016 3:58 PM, PrasannaKumar Chamarty <te...@gmail.com> wrote:



Hi,

What is the best way (in terms of maintenance required with new lucene
releases) to allow splitting of words on "." and "_" for indexing ? Thank
you.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org