You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2007/12/02 16:39:50 UTC
Re: CJK Analyzers for Solr
Wunder - are you aware of any free dictionaries for either C or J or K? When I dealt with this in the past, I looked for something free, but found only commercial dictionaries.
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Walter Underwood <wu...@netflix.com>
To: solr-user@lucene.apache.org
Sent: Wednesday, November 28, 2007 5:43:32 PM
Subject: Re: CJK Analyzers for Solr
With Ultraseek, we switched to a dictionary-based segmenter for Chinese
because the N-gram highlighting wasn't acceptable to our Chinese
customers.
I guess it is something to check for each application.
wunder
On 11/27/07 10:46 PM, "Otis Gospodnetic" <ot...@yahoo.com>
wrote:
> For what it's worth I worked on indexing and searching a *massive*
pile of
> data, a good portion of which was in CJ and some K. The n-gram
approach was
> used for all 3 languages and the quality of search results, including
> highlighting was evaluated and okay-ed by native speakers of these
languages.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
> ----- Original Message ----
> From: Walter Underwood <wu...@netflix.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, November 27, 2007 2:41:38 PM
> Subject: Re: CJK Analyzers for Solr
>
> Dictionaries are surprisingly expensive to build and maintain and
> bi-gram is surprisingly effective for Chinese. See this paper:
>
> http://citeseer.ist.psu.edu/kwok97comparing.html
>
> I expect that n-gram indexing would be less effective for Japanese
> because it is an inflected language. Korean is even harder. It might
> work to break Korean into the phonetic subparts and use n-gram on
> those.
>
> You should not do term highlighting with any of the n-gram methods.
> The relevance can be very good, but the highlighting just looks dumb.
>
> wunder
>
> On 11/27/07 8:54 AM, "Eswar K" <kj...@gmail.com> wrote:
>
>> Is there any specific reason why the CJK analyzers in Solr were
> chosen to be
>> n-gram based instead of it being a morphological analyzer which is
> kind of
>> implemented in Google as it considered to be more effective than the
> n-gram
>> ones?
>>
>> Regards,
>> Eswar
>>
>>
>>
>> On Nov 27, 2007 7:57 AM, Eswar K <kj...@gmail.com> wrote:
>>
>>> thanks james...
>>>
>>> How much time does it take to index 18m docs?
>>>
>>> - Eswar
>>>
>>>
>>> On Nov 27, 2007 7:43 AM, James liu <liuping.james@gmail.com >
wrote:
>>>
>>>> i not use HYLANDA analyzer.
>>>>
>>>> i use je-analyzer and indexing at least 18m docs.
>>>>
>>>> i m sorry i only use chinese analyzer.
>>>>
>>>>
>>>> On Nov 27, 2007 10:01 AM, Eswar K <kj...@gmail.com> wrote:
>>>>
>>>>> What is the performance of these CJK analyzers (one in lucene and
>>>> hylanda
>>>>> )?
>>>>> We would potentially be indexing millions of documents.
>>>>>
>>>>> James,
>>>>>
>>>>> We would have a look at hylanda too. What abt japanese and korean
>>>>> analyzers,
>>>>> any recommendations?
>>>>>
>>>>> - Eswar
>>>>>
>>>>> On Nov 27, 2007 7:21 AM, James liu <li...@gmail.com>
> wrote:
>>>>>
>>>>>> I don't think NGram is good method for Chinese.
>>>>>>
>>>>>> CJKAnalyzer of Lucene is 2-Gram.
>>>>>>
>>>>>> Eswar K:
>>>>>> if it is chinese analyzer,,i recommend
> hylanda（www.hylanda.com）,,,it
>>>> is
>>>>>> the best chinese analyzer and it not free.
>>>>>> if u wanna free chinese analyzer, maybe u can try je-analyzer.
> it
>>>> have
>>>>>> some problem when using it.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Nov 27, 2007 5:56 AM, Otis Gospodnetic <
>>>> otis_gospodnetic@yahoo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Eswar,
>>>>>>>
>>>>>>> We've uses the NGram stuff that exists in Lucene's
>>>> contrib/analyzers
>>>>>>> instead of CJK. Doesn't that allow you to do everything that
> the
>>>>>> Chinese
>>>>>>> and CJK analyzers do? It's been a few months since I've looked
> at
>>>>>> Chinese
>>>>>>> and CJK Analzyers, so I could be off.
>>>>>>>
>>>>>>> Otis
>>>>>>>
>>>>>>> --
>>>>>>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>>>>>>
>>>>>>> ----- Original Message ----
>>>>>>> From: Eswar K <kj...@gmail.com>
>>>>>>> To: solr-user@lucene.apache.org
>>>>>>> Sent: Monday, November 26, 2007 8:30:52 AM
>>>>>>> Subject: CJK Analyzers for Solr
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Does Solr come with Language analyzers for CJK? If not, can you
>>>> please
>>>>>>> direct me to some good CJK analyzers?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Eswar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> regards
>>>>>> jl
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> regards
>>>> jl
>>>>
>>>
>>>
>
>
>
>
Re: CJK Analyzers for Solr
Posted by James liu <li...@gmail.com>.
it seems good.
On Dec 3, 2007 1:01 AM, Ken Krugler <kk...@transpac.com> wrote:
> >Wunder - are you aware of any free dictionaries
> >for either C or J or K? When I dealt with this
> >in the past, I looked for something free, but
> >found only commercial dictionaries.
>
> I would use data files from:
>
> http://ftp.monash.edu.au/pub/nihongo/00INDEX.html
>
> -- Ken
>
>
> >Sematext -- http://sematext.com/ -- Lucene -
> >Solr - Nutch ----- Original Message ---- From:
> >Walter Underwood <wu...@netflix.com> To:
> >solr-user@lucene.apache.org Sent: Wednesday,
> >November 28, 2007 5:43:32 PM Subject: Re: CJK
> >Analyzers for Solr With Ultraseek, we switched
> >to a dictionary-based segmenter for Chinese
> >because the N-gram highlighting wasn't
> >acceptable to our Chinese customers. I guess it
> >is something to check for each application.
> >wunder On 11/27/07 10:46 PM, "Otis Gospodnetic"
> ><ot...@yahoo.com> wrote: > For what
> >it's worth I worked on indexing and searching a
> >*massive* pile of > data, a good portion of
> >which was in CJ and some K. The n-gram approach
> >was > used for all 3 languages and the quality
> >of search results, including > highlighting was
> >evaluated and okay-ed by native speakers of
> >these languages. > > Otis > -- > Sematext --
> >http://sematext.com/ -- Lucene - Solr -
> >Nutch > > ----- Original Message ---- > From:
> >Walter Underwood <wu...@netflix.com> > To:
> >solr-user@lucene.apache.org > Sent: Tuesday,
> >November 27, 2007 2:41:38 PM > Subject: Re: CJK
> >Analyzers for Solr > > Dictionaries are
> >surprisingly expensive to build and maintain
> >and > bi-gram is surprisingly effective for
> >Chinese. See this paper: > >
> >http://citeseer.ist.psu.edu/kwok97comparing.html > >
> >I expect that n-gram indexing would be less
> >effective for Japanese > because it is an
> >inflected language. Korean is even harder. It
> >might > work to break Korean into the phonetic
> >subparts and use n-gram on > those. > > You
> >should not do term highlighting with any of the
> >n-gram methods. > The relevance can be very
> >good, but the highlighting just looks dumb. > >
> >wunder > > On 11/27/07 8:54 AM, "Eswar K"
> ><kj...@gmail.com> wrote: > >> Is there any
> >specific reason why the CJK analyzers in Solr
> >were > chosen to be >> n-gram based instead of
> >it being a morphological analyzer which is >
> >kind of >> implemented in Google as it
> >considered to be more effective than the >
> >n-gram >> ones? >> >> Regards, >>
> >Eswar >> >> >> >> On Nov 27, 2007 7:57 AM, Eswar
> >K <kj...@gmail.com> wrote: >> >>> thanks
> >james... >>> >>> How much time does it take to
> >index 18m docs? >>> >>> - Eswar >>> >>> >>> On
> >Nov 27, 2007 7:43 AM, James liu
> ><liuping.james@gmail.com > wrote: >>> >>>> i not
> >use HYLANDA analyzer. >>>> >>>> i use
> >je-analyzer and indexing at least 18m
> >docs. >>>> >>>> i m sorry i only use chinese
> >analyzer. >>>> >>>> >>>> On Nov 27, 2007 10:01
> >AM, Eswar K <kj...@gmail.com>
> >wrote: >>>> >>>>> What is the performance of
> >these CJK analyzers (one in lucene and >>>>
> >hylanda >>>>> )? >>>>> We would potentially be
> >indexing millions of documents. >>>>> >>>>>
> >James, >>>>> >>>>> We would have a look at
> >hylanda too. What abt japanese and korean >>>>>
> >analyzers, >>>>> any
> >recommendations? >>>>> >>>>> - Eswar >>>>> >>>>>
> >On Nov 27, 2007 7:21 AM, James liu
> ><li...@gmail.com> > wrote: >>>>> >>>>>>
> >I don't think NGram is good method for
> >Chinese. >>>>>> >>>>>> CJKAnalyzer of Lucene is
> >2-Gram. >>>>>> >>>>>> Eswar K: >>>>>> if it is
> >chinese analyzer,,i recommend >
> >hylandaÅiwww.hylanda.comÅj,,,it >>>> is >>>>>>
> >the best chinese analyzer and it not
> >free. >>>>>> if u wanna free chinese analyzer,
> >maybe u can try je-analyzer. > it >>>>
> >have >>>>>> some problem when using
> >it. >>>>>> >>>>>> >>>>>> >>>>>> On Nov 27, 2007
> >5:56 AM, Otis Gospodnetic < >>>>
> >otis_gospodnetic@yahoo.com> >>>>>>
> >wrote: >>>>>> >>>>>>> Eswar, >>>>>>> >>>>>>>
> >We've uses the NGram stuff that exists in
> >Lucene's >>>> contrib/analyzers >>>>>>> instead
> >of CJK. Doesn't that allow you to do everything
> >that > the >>>>>> Chinese >>>>>>> and CJK
> >analyzers do? It's been a few months since I've
> >looked > at >>>>>> Chinese >>>>>>> and CJK
> >Analzyers, so I could be off. >>>>>>> >>>>>>>
> >Otis >>>>>>> >>>>>>> -- >>>>>>> Sematext --
> >http://sematext.com/ -- Lucene - Solr -
> >Nutch >>>>>>> >>>>>>> ----- Original Message
> >---- >>>>>>> From: Eswar K
> ><kj...@gmail.com> >>>>>>> To:
> >solr-user@lucene.apache.org >>>>>>> Sent:
> >Monday, November 26, 2007 8:30:52 AM >>>>>>>
> >Subject: CJK Analyzers for Solr >>>>>>> >>>>>>>
> >Hi, >>>>>>> >>>>>>> Does Solr come with Language
> >analyzers for CJK? If not, can you >>>>
> >please >>>>>>> direct me to some good CJK
> >analyzers? >>>>>>> >>>>>>> Regards, >>>>>>>
> >Eswar >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>>
> >-- >>>>>> regards >>>>>>
> >jl >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>>
> >regards >>>> jl >>>> >>> >>> > > > >
>
>
> --
> Ken Krugler
> Krugle, Inc.
> +1 530-210-6378
> "If you can't find it, you can't fix it"
--
regards
jl
Re: CJK Analyzers for Solr
Posted by Ken Krugler <kk...@transpac.com>.
>Wunder - are you aware of any free dictionaries
>for either C or J or K? When I dealt with this
>in the past, I looked for something free, but
>found only commercial dictionaries.
I would use data files from:
http://ftp.monash.edu.au/pub/nihongo/00INDEX.html
-- Ken
>Sematext -- http://sematext.com/ -- Lucene -
>Solr - Nutch ----- Original Message ---- From:
>Walter Underwood <wu...@netflix.com> To:
>solr-user@lucene.apache.org Sent: Wednesday,
>November 28, 2007 5:43:32 PM Subject: Re: CJK
>Analyzers for Solr With Ultraseek, we switched
>to a dictionary-based segmenter for Chinese
>because the N-gram highlighting wasn't
>acceptable to our Chinese customers. I guess it
>is something to check for each application.
>wunder On 11/27/07 10:46 PM, "Otis Gospodnetic"
><ot...@yahoo.com> wrote: > For what
>it's worth I worked on indexing and searching a
>*massive* pile of > data, a good portion of
>which was in CJ and some K. The n-gram approach
>was > used for all 3 languages and the quality
>of search results, including > highlighting was
>evaluated and okay-ed by native speakers of
>these languages. > > Otis > -- > Sematext --
>http://sematext.com/ -- Lucene - Solr -
>Nutch > > ----- Original Message ---- > From:
>Walter Underwood <wu...@netflix.com> > To:
>solr-user@lucene.apache.org > Sent: Tuesday,
>November 27, 2007 2:41:38 PM > Subject: Re: CJK
>Analyzers for Solr > > Dictionaries are
>surprisingly expensive to build and maintain
>and > bi-gram is surprisingly effective for
>Chinese. See this paper: > >
>http://citeseer.ist.psu.edu/kwok97comparing.html > >
>I expect that n-gram indexing would be less
>effective for Japanese > because it is an
>inflected language. Korean is even harder. It
>might > work to break Korean into the phonetic
>subparts and use n-gram on > those. > > You
>should not do term highlighting with any of the
>n-gram methods. > The relevance can be very
>good, but the highlighting just looks dumb. > >
>wunder > > On 11/27/07 8:54 AM, "Eswar K"
><kj...@gmail.com> wrote: > >> Is there any
>specific reason why the CJK analyzers in Solr
>were > chosen to be >> n-gram based instead of
>it being a morphological analyzer which is >
>kind of >> implemented in Google as it
>considered to be more effective than the >
>n-gram >> ones? >> >> Regards, >>
>Eswar >> >> >> >> On Nov 27, 2007 7:57 AM, Eswar
>K <kj...@gmail.com> wrote: >> >>> thanks
>james... >>> >>> How much time does it take to
>index 18m docs? >>> >>> - Eswar >>> >>> >>> On
>Nov 27, 2007 7:43 AM, James liu
><liuping.james@gmail.com > wrote: >>> >>>> i not
>use HYLANDA analyzer. >>>> >>>> i use
>je-analyzer and indexing at least 18m
>docs. >>>> >>>> i m sorry i only use chinese
>analyzer. >>>> >>>> >>>> On Nov 27, 2007 10:01
>AM, Eswar K <kj...@gmail.com>
>wrote: >>>> >>>>> What is the performance of
>these CJK analyzers (one in lucene and >>>>
>hylanda >>>>> )? >>>>> We would potentially be
>indexing millions of documents. >>>>> >>>>>
>James, >>>>> >>>>> We would have a look at
>hylanda too. What abt japanese and korean >>>>>
>analyzers, >>>>> any
>recommendations? >>>>> >>>>> - Eswar >>>>> >>>>>
>On Nov 27, 2007 7:21 AM, James liu
><li...@gmail.com> > wrote: >>>>> >>>>>>
>I don't think NGram is good method for
>Chinese. >>>>>> >>>>>> CJKAnalyzer of Lucene is
>2-Gram. >>>>>> >>>>>> Eswar K: >>>>>> if it is
>chinese analyzer,,i recommend >
>hylandaÅiwww.hylanda.comÅj,,,it >>>> is >>>>>>
>the best chinese analyzer and it not
>free. >>>>>> if u wanna free chinese analyzer,
>maybe u can try je-analyzer. > it >>>>
>have >>>>>> some problem when using
>it. >>>>>> >>>>>> >>>>>> >>>>>> On Nov 27, 2007
>5:56 AM, Otis Gospodnetic < >>>>
>otis_gospodnetic@yahoo.com> >>>>>>
>wrote: >>>>>> >>>>>>> Eswar, >>>>>>> >>>>>>>
>We've uses the NGram stuff that exists in
>Lucene's >>>> contrib/analyzers >>>>>>> instead
>of CJK. Doesn't that allow you to do everything
>that > the >>>>>> Chinese >>>>>>> and CJK
>analyzers do? It's been a few months since I've
>looked > at >>>>>> Chinese >>>>>>> and CJK
>Analzyers, so I could be off. >>>>>>> >>>>>>>
>Otis >>>>>>> >>>>>>> -- >>>>>>> Sematext --
>http://sematext.com/ -- Lucene - Solr -
>Nutch >>>>>>> >>>>>>> ----- Original Message
>---- >>>>>>> From: Eswar K
><kj...@gmail.com> >>>>>>> To:
>solr-user@lucene.apache.org >>>>>>> Sent:
>Monday, November 26, 2007 8:30:52 AM >>>>>>>
>Subject: CJK Analyzers for Solr >>>>>>> >>>>>>>
>Hi, >>>>>>> >>>>>>> Does Solr come with Language
>analyzers for CJK? If not, can you >>>>
>please >>>>>>> direct me to some good CJK
>analyzers? >>>>>>> >>>>>>> Regards, >>>>>>>
>Eswar >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>>
>-- >>>>>> regards >>>>>>
>jl >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>>
>regards >>>> jl >>>> >>> >>> > > > >
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"If you can't find it, you can't fix it"