You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@lucenenet.apache.org by Markus Doerig <fa...@gmail.com> on 2010/05/06 15:42:31 UTC

Unwanted Tokens

I'm trying to write my own Filter (using StandardFilter) as a template...

Most of the cases i could solve with adding my own StopWords, but some cases
of Terms i want to avoid,
i guess i need to write my own Filter.

here what i wrote so far:

        public override Lucene.Net.Analysis.Token Next()
        {
            Lucene.Net.Analysis.Token t = input.Next();

            if (t == null)
                return null;

            System.String text = t.TermText();
            System.String type = t.Type();



            if (text.Length == 1)  // Remove all Terms with only one char
                return input.Next();
            else
                return t;
      }


When i check in Luke, it seems that some one-char terms are removed ,but not
all.
I'm doing something wrong?

Thanks for any help.
Markus.

Re: Unwanted Tokens

Posted by Markus Doerig <fa...@gmail.com>.

Thanks alot for the Help, this was the right pointer!

   this.Next().

It works now like i hoped (it's my first use of Lucene.Net)

I just discovered that i was working with the 2.0 version, now i'm tuning my
code to 2.91
(where .Next is obsolete ,but still working)

Cheers,
Markus.

RE: Unwanted Tokens

Posted by Digy <di...@gmail.com>.

For input "This is a b text", your code would output [this] [is] [b] [text].
Call "return this.Next()" instead of "return input.Next()"

DIGY

-----Original Message-----
From: Markus Doerig [mailto:farangkao@gmail.com] 
Sent: Thursday, May 06, 2010 4:43 PM
To: lucene-net-user@lucene.apache.org
Subject: Unwanted Tokens

I'm trying to write my own Filter (using StandardFilter) as a template...

Most of the cases i could solve with adding my own StopWords, but some cases
of Terms i want to avoid,
i guess i need to write my own Filter.

here what i wrote so far:

        public override Lucene.Net.Analysis.Token Next()
        {
            Lucene.Net.Analysis.Token t = input.Next();

            if (t == null)
                return null;

            System.String text = t.TermText();
            System.String type = t.Type();

            if (text.Length == 1)  // Remove all Terms with only one char
                return input.Next();
            else
                return t;
      }

When i check in Luke, it seems that some one-char terms are removed ,but not
all.
I'm doing something wrong?

Thanks for any help.
Markus.

RE: Unwanted Tokens

Posted by Heath Aldrich <ha...@aes2.com>.

Do you need to trim off the text?
Is it possible that the length isn't 1 because of some non printable character (tab, crlf, etc)

-----Original Message-----
From: Digy [mailto:digydigy@gmail.com] 
Sent: Thursday, May 06, 2010 10:29 AM
To: lucene-net-user@lucene.apache.org
Subject: RE: Unwanted Tokens

>>>>> return input.Next(); 
How can you can be sure that input.Next() is not a one-char token?

DIGY

-----Original Message-----
From: Markus Doerig [mailto:farangkao@gmail.com] 
Sent: Thursday, May 06, 2010 4:43 PM
To: lucene-net-user@lucene.apache.org
Subject: Unwanted Tokens

I'm trying to write my own Filter (using StandardFilter) as a template...

Most of the cases i could solve with adding my own StopWords, but some cases
of Terms i want to avoid,
i guess i need to write my own Filter.

here what i wrote so far:

        public override Lucene.Net.Analysis.Token Next()
        {
            Lucene.Net.Analysis.Token t = input.Next();

            if (t == null)
                return null;

            System.String text = t.TermText();
            System.String type = t.Type();

            if (text.Length == 1)  // Remove all Terms with only one char
                return input.Next();
            else
                return t;
      }

When i check in Luke, it seems that some one-char terms are removed ,but not
all.
I'm doing something wrong?

Thanks for any help.
Markus.

RE: Unwanted Tokens

Posted by Digy <di...@gmail.com>.

>>>>> return input.Next(); 
How can you can be sure that input.Next() is not a one-char token?

DIGY

-----Original Message-----
From: Markus Doerig [mailto:farangkao@gmail.com] 
Sent: Thursday, May 06, 2010 4:43 PM
To: lucene-net-user@lucene.apache.org
Subject: Unwanted Tokens

I'm trying to write my own Filter (using StandardFilter) as a template...

Most of the cases i could solve with adding my own StopWords, but some cases
of Terms i want to avoid,
i guess i need to write my own Filter.

here what i wrote so far:

        public override Lucene.Net.Analysis.Token Next()
        {
            Lucene.Net.Analysis.Token t = input.Next();

            if (t == null)
                return null;

            System.String text = t.TermText();
            System.String type = t.Type();

            if (text.Length == 1)  // Remove all Terms with only one char
                return input.Next();
            else
                return t;
      }

When i check in Luke, it seems that some one-char terms are removed ,but not
all.
I'm doing something wrong?

Thanks for any help.
Markus.