You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ctakes.apache.org by "Miller, Timothy" <Ti...@childrens.harvard.edu> on 2014/04/17 18:27:22 UTC
lvg entries
The LVG annotator creates an enormous number of "lemmas" for every
WordToken in the CAS, and I'm wondering what the original purpose was? I
think this is probably a minor bottleneck for speed but mostly a pretty
big space hog (at least 50% of the space of xmi files in my tests).
As of right now I'm not sure if any downstream components are using
these lemmas, and on a manual inspection the precision seems to be
pretty abysmal (meaning most of them are nonsensical as lexical
variants), so as I said, just wondering if we can revisit why cTAKES
generates so many and whether that component can be optimized.
Thanks
Tim
Re: lvg entries
Posted by andy mcmurry <mc...@gmail.com>.
+1 false ... I think
I just wonder what side effects there might be to tweaking LVG
On Fri, Apr 18, 2014 at 11:56 AM, Finan, Sean <
Sean.Finan@childrens.harvard.edu> wrote:
> +1 false
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Friday, April 18, 2014 2:54 PM
> To: dev@ctakes.apache.org
> Subject: Re: lvg entries
>
> Thanks for tracking that down Andy.
>
> I am making a pass at UimaFit-izing the configuration parameters for all
> the annotators in the default pipeline, before I create the static factory
> methods like we recently discussed. Should I go ahead and change this to
> make default behavior be false?
>
> Tim
>
>
> On 04/18/2014 12:47 AM, andy mcmurry wrote:
> > There is a lot of config handling, maybe PostLemmas is being set to
> > true or
> > configInit() is not setting up the NLM wrapper incorrectly.
> >
> > ctakes-lvg *README*
> > Note: as distributed, PostLemmas is set to false. This is done to
> > reduce the size of the CAS.
> > Set PostLemmas to true to have org.apache.ctakes.typesystem.type.Lemma
> > annotations added to the CAS.
> >
> > *LvgAnnotator.xml *
> > PostLemmas = True
> >
> > *LvgAnnotator.java*
> > if (postLemmas) {
> > lvgResource.getLvgLex()
> > }
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 17, 2014 at 3:23 PM, Masanz, James J. <Masanz.James@mayo.edu
> >wrote:
> >
> >> The normalizedForm field is filled in. It is used by dictionary lookup.
> >>
> >> So, for example, if the dictionary would contain "lymph node" but not
> >> "lymph nodes", a document with text of "lymph nodes" would match the
> >> dictionary entry "lymph node" because "node", being the normalized
> >> form of "nodes", would be used when searching dictionary entries (in
> >> addition to searching dictionary entries for "nodes")
> >>
> >> -----Original Message-----
> >> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> >> Sent: Thursday, April 17, 2014 4:33 PM
> >> To: dev@ctakes.apache.org
> >> Subject: Re: lvg entries
> >>
> >> Quick follow-up since I was interested. The current dependency parser
> >> does have the option to use ctakes lemmas or do its own lemmatizing,
> >> but that doesn't use the lemma field, it uses the normalizedForm
> >> field. I'm not sure if that field is actually ever filled in -- on my
> >> example data it is always null.
> >>
> >> Tim
> >>
> >> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> >>> Offhand I recall at least one of the dependency parsers used the
> >>> Lemma
> >> annotations at one point.
> >>> Not sure if still does.
> >>>
> >>> There is an option for turning off the posting of the lemmas to the
> cas.
> >>>
> >>> Hope that helps
> >>>
> >>> -----Original Message-----
> >>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> >>> Sent: Thursday, April 17, 2014 11:27 AM
> >>> To: dev@ctakes.apache.org
> >>> Subject: lvg entries
> >>>
> >>> The LVG annotator creates an enormous number of "lemmas" for every
> >>> WordToken in the CAS, and I'm wondering what the original purpose
> >>> was? I think this is probably a minor bottleneck for speed but
> >>> mostly a pretty big space hog (at least 50% of the space of xmi files
> in my tests).
> >>>
> >>> As of right now I'm not sure if any downstream components are using
> >>> these lemmas, and on a manual inspection the precision seems to be
> >>> pretty abysmal (meaning most of them are nonsensical as lexical
> >>> variants), so as I said, just wondering if we can revisit why cTAKES
> >>> generates so many and whether that component can be optimized.
> >>>
> >>> Thanks
> >>> Tim
> >>>
> >>>
> >>
>
>
RE: lvg entries
Posted by "Finan, Sean" <Se...@childrens.harvard.edu>.
+1 false
-----Original Message-----
From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
Sent: Friday, April 18, 2014 2:54 PM
To: dev@ctakes.apache.org
Subject: Re: lvg entries
Thanks for tracking that down Andy.
I am making a pass at UimaFit-izing the configuration parameters for all the annotators in the default pipeline, before I create the static factory methods like we recently discussed. Should I go ahead and change this to make default behavior be false?
Tim
On 04/18/2014 12:47 AM, andy mcmurry wrote:
> There is a lot of config handling, maybe PostLemmas is being set to
> true or
> configInit() is not setting up the NLM wrapper incorrectly.
>
> ctakes-lvg *README*
> Note: as distributed, PostLemmas is set to false. This is done to
> reduce the size of the CAS.
> Set PostLemmas to true to have org.apache.ctakes.typesystem.type.Lemma
> annotations added to the CAS.
>
> *LvgAnnotator.xml *
> PostLemmas = True
>
> *LvgAnnotator.java*
> if (postLemmas) {
> lvgResource.getLvgLex()
> }
>
>
>
>
>
>
>
> On Thu, Apr 17, 2014 at 3:23 PM, Masanz, James J. <Ma...@mayo.edu>wrote:
>
>> The normalizedForm field is filled in. It is used by dictionary lookup.
>>
>> So, for example, if the dictionary would contain "lymph node" but not
>> "lymph nodes", a document with text of "lymph nodes" would match the
>> dictionary entry "lymph node" because "node", being the normalized
>> form of "nodes", would be used when searching dictionary entries (in
>> addition to searching dictionary entries for "nodes")
>>
>> -----Original Message-----
>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>> Sent: Thursday, April 17, 2014 4:33 PM
>> To: dev@ctakes.apache.org
>> Subject: Re: lvg entries
>>
>> Quick follow-up since I was interested. The current dependency parser
>> does have the option to use ctakes lemmas or do its own lemmatizing,
>> but that doesn't use the lemma field, it uses the normalizedForm
>> field. I'm not sure if that field is actually ever filled in -- on my
>> example data it is always null.
>>
>> Tim
>>
>> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
>>> Offhand I recall at least one of the dependency parsers used the
>>> Lemma
>> annotations at one point.
>>> Not sure if still does.
>>>
>>> There is an option for turning off the posting of the lemmas to the cas.
>>>
>>> Hope that helps
>>>
>>> -----Original Message-----
>>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>>> Sent: Thursday, April 17, 2014 11:27 AM
>>> To: dev@ctakes.apache.org
>>> Subject: lvg entries
>>>
>>> The LVG annotator creates an enormous number of "lemmas" for every
>>> WordToken in the CAS, and I'm wondering what the original purpose
>>> was? I think this is probably a minor bottleneck for speed but
>>> mostly a pretty big space hog (at least 50% of the space of xmi files in my tests).
>>>
>>> As of right now I'm not sure if any downstream components are using
>>> these lemmas, and on a manual inspection the precision seems to be
>>> pretty abysmal (meaning most of them are nonsensical as lexical
>>> variants), so as I said, just wondering if we can revisit why cTAKES
>>> generates so many and whether that component can be optimized.
>>>
>>> Thanks
>>> Tim
>>>
>>>
>>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Thanks for tracking that down Andy.
I am making a pass at UimaFit-izing the configuration parameters for all
the annotators in the default pipeline, before I create the static
factory methods like we recently discussed. Should I go ahead and change
this to make default behavior be false?
Tim
On 04/18/2014 12:47 AM, andy mcmurry wrote:
> There is a lot of config handling, maybe PostLemmas is being set to true or
> configInit() is not setting up the NLM wrapper incorrectly.
>
> ctakes-lvg *README*
> Note: as distributed, PostLemmas is set to false. This is done to reduce
> the size of the CAS.
> Set PostLemmas to true to have org.apache.ctakes.typesystem.type.Lemma
> annotations added to the CAS.
>
> *LvgAnnotator.xml *
> PostLemmas = True
>
> *LvgAnnotator.java*
> if (postLemmas) {
> lvgResource.getLvgLex()
> }
>
>
>
>
>
>
>
> On Thu, Apr 17, 2014 at 3:23 PM, Masanz, James J. <Ma...@mayo.edu>wrote:
>
>> The normalizedForm field is filled in. It is used by dictionary lookup.
>>
>> So, for example, if the dictionary would contain "lymph node" but not
>> "lymph nodes", a document with text of "lymph nodes" would match the
>> dictionary entry "lymph node" because "node", being the normalized form of
>> "nodes", would be used when searching dictionary entries (in addition to
>> searching dictionary entries for "nodes")
>>
>> -----Original Message-----
>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>> Sent: Thursday, April 17, 2014 4:33 PM
>> To: dev@ctakes.apache.org
>> Subject: Re: lvg entries
>>
>> Quick follow-up since I was interested. The current dependency parser
>> does have the option to use ctakes lemmas or do its own lemmatizing, but
>> that doesn't use the lemma field, it uses the normalizedForm field. I'm
>> not sure if that field is actually ever filled in -- on my example data
>> it is always null.
>>
>> Tim
>>
>> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
>>> Offhand I recall at least one of the dependency parsers used the Lemma
>> annotations at one point.
>>> Not sure if still does.
>>>
>>> There is an option for turning off the posting of the lemmas to the cas.
>>>
>>> Hope that helps
>>>
>>> -----Original Message-----
>>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>>> Sent: Thursday, April 17, 2014 11:27 AM
>>> To: dev@ctakes.apache.org
>>> Subject: lvg entries
>>>
>>> The LVG annotator creates an enormous number of "lemmas" for every
>>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>>> think this is probably a minor bottleneck for speed but mostly a pretty
>>> big space hog (at least 50% of the space of xmi files in my tests).
>>>
>>> As of right now I'm not sure if any downstream components are using
>>> these lemmas, and on a manual inspection the precision seems to be
>>> pretty abysmal (meaning most of them are nonsensical as lexical
>>> variants), so as I said, just wondering if we can revisit why cTAKES
>>> generates so many and whether that component can be optimized.
>>>
>>> Thanks
>>> Tim
>>>
>>>
>>
Re: lvg entries
Posted by andy mcmurry <mc...@gmail.com>.
There is a lot of config handling, maybe PostLemmas is being set to true or
configInit() is not setting up the NLM wrapper incorrectly.
ctakes-lvg *README*
Note: as distributed, PostLemmas is set to false. This is done to reduce
the size of the CAS.
Set PostLemmas to true to have org.apache.ctakes.typesystem.type.Lemma
annotations added to the CAS.
*LvgAnnotator.xml *
PostLemmas = True
*LvgAnnotator.java*
if (postLemmas) {
lvgResource.getLvgLex()
}
On Thu, Apr 17, 2014 at 3:23 PM, Masanz, James J. <Ma...@mayo.edu>wrote:
> The normalizedForm field is filled in. It is used by dictionary lookup.
>
> So, for example, if the dictionary would contain "lymph node" but not
> "lymph nodes", a document with text of "lymph nodes" would match the
> dictionary entry "lymph node" because "node", being the normalized form of
> "nodes", would be used when searching dictionary entries (in addition to
> searching dictionary entries for "nodes")
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 4:33 PM
> To: dev@ctakes.apache.org
> Subject: Re: lvg entries
>
> Quick follow-up since I was interested. The current dependency parser
> does have the option to use ctakes lemmas or do its own lemmatizing, but
> that doesn't use the lemma field, it uses the normalizedForm field. I'm
> not sure if that field is actually ever filled in -- on my example data
> it is always null.
>
> Tim
>
> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> > Offhand I recall at least one of the dependency parsers used the Lemma
> annotations at one point.
> > Not sure if still does.
> >
> > There is an option for turning off the posting of the lemmas to the cas.
> >
> > Hope that helps
> >
> > -----Original Message-----
> > From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> > Sent: Thursday, April 17, 2014 11:27 AM
> > To: dev@ctakes.apache.org
> > Subject: lvg entries
> >
> > The LVG annotator creates an enormous number of "lemmas" for every
> > WordToken in the CAS, and I'm wondering what the original purpose was? I
> > think this is probably a minor bottleneck for speed but mostly a pretty
> > big space hog (at least 50% of the space of xmi files in my tests).
> >
> > As of right now I'm not sure if any downstream components are using
> > these lemmas, and on a manual inspection the precision seems to be
> > pretty abysmal (meaning most of them are nonsensical as lexical
> > variants), so as I said, just wondering if we can revisit why cTAKES
> > generates so many and whether that component can be optimized.
> >
> > Thanks
> > Tim
> >
> >
>
>
RE: lvg entries
Posted by "Masanz, James J." <Ma...@mayo.edu>.
You are right, I was thinking of the field called canonicalForm.
normlizedForm is set by ExtractionPrepAnnotator.java - but if I remember right, that's at the end of the pipelines that it's included in. And it's set to either the canonicalForm (if there is one) or the coveredText
Not sure what the intent there was.
-----Original Message-----
From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
Sent: Friday, April 18, 2014 11:16 AM
To: dev@ctakes.apache.org
Subject: Re: lvg entries
Hmm... I don't see normalizedForm filled in. I see LVG filling in
canonicalForm, is it possible that's what you're thinking of? (Not that
I know what the difference is or is supposed to be, just going off what
I see in my xmis.)
Tim
On 04/17/2014 06:23 PM, Masanz, James J. wrote:
> The normalizedForm field is filled in. It is used by dictionary lookup.
>
> So, for example, if the dictionary would contain "lymph node" but not "lymph nodes", a document with text of "lymph nodes" would match the dictionary entry "lymph node" because "node", being the normalized form of "nodes", would be used when searching dictionary entries (in addition to searching dictionary entries for "nodes")
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 4:33 PM
> To: dev@ctakes.apache.org
> Subject: Re: lvg entries
>
> Quick follow-up since I was interested. The current dependency parser
> does have the option to use ctakes lemmas or do its own lemmatizing, but
> that doesn't use the lemma field, it uses the normalizedForm field. I'm
> not sure if that field is actually ever filled in -- on my example data
> it is always null.
>
> Tim
>
> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
>> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
>> Not sure if still does.
>>
>> There is an option for turning off the posting of the lemmas to the cas.
>>
>> Hope that helps
>>
>> -----Original Message-----
>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>> Sent: Thursday, April 17, 2014 11:27 AM
>> To: dev@ctakes.apache.org
>> Subject: lvg entries
>>
>> The LVG annotator creates an enormous number of "lemmas" for every
>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>> think this is probably a minor bottleneck for speed but mostly a pretty
>> big space hog (at least 50% of the space of xmi files in my tests).
>>
>> As of right now I'm not sure if any downstream components are using
>> these lemmas, and on a manual inspection the precision seems to be
>> pretty abysmal (meaning most of them are nonsensical as lexical
>> variants), so as I said, just wondering if we can revisit why cTAKES
>> generates so many and whether that component can be optimized.
>>
>> Thanks
>> Tim
>>
>>
>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Hmm... I don't see normalizedForm filled in. I see LVG filling in
canonicalForm, is it possible that's what you're thinking of? (Not that
I know what the difference is or is supposed to be, just going off what
I see in my xmis.)
Tim
On 04/17/2014 06:23 PM, Masanz, James J. wrote:
> The normalizedForm field is filled in. It is used by dictionary lookup.
>
> So, for example, if the dictionary would contain "lymph node" but not "lymph nodes", a document with text of "lymph nodes" would match the dictionary entry "lymph node" because "node", being the normalized form of "nodes", would be used when searching dictionary entries (in addition to searching dictionary entries for "nodes")
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 4:33 PM
> To: dev@ctakes.apache.org
> Subject: Re: lvg entries
>
> Quick follow-up since I was interested. The current dependency parser
> does have the option to use ctakes lemmas or do its own lemmatizing, but
> that doesn't use the lemma field, it uses the normalizedForm field. I'm
> not sure if that field is actually ever filled in -- on my example data
> it is always null.
>
> Tim
>
> On 04/17/2014 01:57 PM, Masanz, James J. wrote:
>> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
>> Not sure if still does.
>>
>> There is an option for turning off the posting of the lemmas to the cas.
>>
>> Hope that helps
>>
>> -----Original Message-----
>> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
>> Sent: Thursday, April 17, 2014 11:27 AM
>> To: dev@ctakes.apache.org
>> Subject: lvg entries
>>
>> The LVG annotator creates an enormous number of "lemmas" for every
>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>> think this is probably a minor bottleneck for speed but mostly a pretty
>> big space hog (at least 50% of the space of xmi files in my tests).
>>
>> As of right now I'm not sure if any downstream components are using
>> these lemmas, and on a manual inspection the precision seems to be
>> pretty abysmal (meaning most of them are nonsensical as lexical
>> variants), so as I said, just wondering if we can revisit why cTAKES
>> generates so many and whether that component can be optimized.
>>
>> Thanks
>> Tim
>>
>>
>
RE: lvg entries
Posted by "Masanz, James J." <Ma...@mayo.edu>.
The normalizedForm field is filled in. It is used by dictionary lookup.
So, for example, if the dictionary would contain "lymph node" but not "lymph nodes", a document with text of "lymph nodes" would match the dictionary entry "lymph node" because "node", being the normalized form of "nodes", would be used when searching dictionary entries (in addition to searching dictionary entries for "nodes")
-----Original Message-----
From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
Sent: Thursday, April 17, 2014 4:33 PM
To: dev@ctakes.apache.org
Subject: Re: lvg entries
Quick follow-up since I was interested. The current dependency parser
does have the option to use ctakes lemmas or do its own lemmatizing, but
that doesn't use the lemma field, it uses the normalizedForm field. I'm
not sure if that field is actually ever filled in -- on my example data
it is always null.
Tim
On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
> Not sure if still does.
>
> There is an option for turning off the posting of the lemmas to the cas.
>
> Hope that helps
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 11:27 AM
> To: dev@ctakes.apache.org
> Subject: lvg entries
>
> The LVG annotator creates an enormous number of "lemmas" for every
> WordToken in the CAS, and I'm wondering what the original purpose was? I
> think this is probably a minor bottleneck for speed but mostly a pretty
> big space hog (at least 50% of the space of xmi files in my tests).
>
> As of right now I'm not sure if any downstream components are using
> these lemmas, and on a manual inspection the precision seems to be
> pretty abysmal (meaning most of them are nonsensical as lexical
> variants), so as I said, just wondering if we can revisit why cTAKES
> generates so many and whether that component can be optimized.
>
> Thanks
> Tim
>
>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Quick follow-up since I was interested. The current dependency parser
does have the option to use ctakes lemmas or do its own lemmatizing, but
that doesn't use the lemma field, it uses the normalizedForm field. I'm
not sure if that field is actually ever filled in -- on my example data
it is always null.
Tim
On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
> Not sure if still does.
>
> There is an option for turning off the posting of the lemmas to the cas.
>
> Hope that helps
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 11:27 AM
> To: dev@ctakes.apache.org
> Subject: lvg entries
>
> The LVG annotator creates an enormous number of "lemmas" for every
> WordToken in the CAS, and I'm wondering what the original purpose was? I
> think this is probably a minor bottleneck for speed but mostly a pretty
> big space hog (at least 50% of the space of xmi files in my tests).
>
> As of right now I'm not sure if any downstream components are using
> these lemmas, and on a manual inspection the precision seems to be
> pretty abysmal (meaning most of them are nonsensical as lexical
> variants), so as I said, just wondering if we can revisit why cTAKES
> generates so many and whether that component can be optimized.
>
> Thanks
> Tim
>
>
RE: lvg entries
Posted by "Masanz, James J." <Ma...@mayo.edu>.
Before the switch to OpenNLP (which was done before the first opensource release of cTAKES), I believe the Lemma annotations were used by the POS tagger and/or phrasal parser. As far as I know, that was the original intention of the Lemmas. I believe they were turned off by default for some releases, until someone started to use them (or at least look at maybe using them)
That's all just from memory. We'd have to look through histories to see when things changed.
I don't think the Lemma annotations were ever used for dictionary lookup. That used the (single) output of the normalizer function of the LVG component
-----Original Message-----
From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
Sent: Thursday, April 17, 2014 3:34 PM
To: dev@ctakes.apache.org
Subject: Re: lvg entries
Thanks James. Does it ring a bell to you that the original intention was
something like query expansion for a dictionary lookup?
Tim
On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
> Not sure if still does.
>
> There is an option for turning off the posting of the lemmas to the cas.
>
> Hope that helps
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 11:27 AM
> To: dev@ctakes.apache.org
> Subject: lvg entries
>
> The LVG annotator creates an enormous number of "lemmas" for every
> WordToken in the CAS, and I'm wondering what the original purpose was? I
> think this is probably a minor bottleneck for speed but mostly a pretty
> big space hog (at least 50% of the space of xmi files in my tests).
>
> As of right now I'm not sure if any downstream components are using
> these lemmas, and on a manual inspection the precision seems to be
> pretty abysmal (meaning most of them are nonsensical as lexical
> variants), so as I said, just wondering if we can revisit why cTAKES
> generates so many and whether that component can be optimized.
>
> Thanks
> Tim
>
>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Thanks James. Does it ring a bell to you that the original intention was
something like query expansion for a dictionary lookup?
Tim
On 04/17/2014 01:57 PM, Masanz, James J. wrote:
> Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
> Not sure if still does.
>
> There is an option for turning off the posting of the lemmas to the cas.
>
> Hope that helps
>
> -----Original Message-----
> From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
> Sent: Thursday, April 17, 2014 11:27 AM
> To: dev@ctakes.apache.org
> Subject: lvg entries
>
> The LVG annotator creates an enormous number of "lemmas" for every
> WordToken in the CAS, and I'm wondering what the original purpose was? I
> think this is probably a minor bottleneck for speed but mostly a pretty
> big space hog (at least 50% of the space of xmi files in my tests).
>
> As of right now I'm not sure if any downstream components are using
> these lemmas, and on a manual inspection the precision seems to be
> pretty abysmal (meaning most of them are nonsensical as lexical
> variants), so as I said, just wondering if we can revisit why cTAKES
> generates so many and whether that component can be optimized.
>
> Thanks
> Tim
>
>
RE: lvg entries
Posted by "Masanz, James J." <Ma...@mayo.edu>.
Offhand I recall at least one of the dependency parsers used the Lemma annotations at one point.
Not sure if still does.
There is an option for turning off the posting of the lemmas to the cas.
Hope that helps
-----Original Message-----
From: Miller, Timothy [mailto:Timothy.Miller@childrens.harvard.edu]
Sent: Thursday, April 17, 2014 11:27 AM
To: dev@ctakes.apache.org
Subject: lvg entries
The LVG annotator creates an enormous number of "lemmas" for every
WordToken in the CAS, and I'm wondering what the original purpose was? I
think this is probably a minor bottleneck for speed but mostly a pretty
big space hog (at least 50% of the space of xmi files in my tests).
As of right now I'm not sure if any downstream components are using
these lemmas, and on a manual inspection the precision seems to be
pretty abysmal (meaning most of them are nonsensical as lexical
variants), so as I said, just wondering if we can revisit why cTAKES
generates so many and whether that component can be optimized.
Thanks
Tim
RE: lvg entries
Posted by "Finan, Sean" <Se...@childrens.harvard.edu>.
Those variants are not used by the dictionary lookup. I did look at them to see if it was worthwhile for the new dictionary, but they are all over the place so I passed.
________________________________________
From: Miller, Timothy [Timothy.Miller@childrens.harvard.edu]
Sent: Thursday, April 17, 2014 1:25 PM
To: dev@ctakes.apache.org
Subject: Re: lvg entries
Pei and I had a similar discussion in person -- mapping from lexical
variants to a stem might be useful. Pei also mentioned that one intended
use might have been searching the dictionary with lexical variants, but
I don't think that is done. Looking at the precision of the variants, I
think its highly unlikely the speed tradeoff would be worth any
improvements in recall.
Finally, at least in eclipse doing a search on references to the method
to retrieve the lemma entries turns up nothing.
Tim
On 04/17/2014 01:14 PM, Dligach, Dmitriy wrote:
> I don’t know of any applications within cTAKES that make use of this… The reverse (mapping from these “variants” to the normal form) may be useful though.
>
> Dima
>
>
>
>
> On Apr 17, 2014, at 11:50, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>
>> Sure, just as an example, I gave it a note with about 1000 words. It
>> generates 11500 NonEmptyFSList elements (each is basically one lexical
>> variant).
>>
>> For the word "symptomatic", these are the first 10 of 20 lexical variants:
>> Symptomaticer/JJ
>> Symptomaticer/RB
>> Symptomaticed/VB
>> Symptomaticcing/VB
>> Symptomatics/VB
>> Symptomatics/NN
>> Symptomaticked/VB
>> Symptomatic/VB
>> Symptomatic/JJ
>> Symptomatic/RB
>>
>> Tim
>>
>>
>> On 04/17/2014 12:31 PM, Dligach, Dmitriy wrote:
>>> Tim, this is a very interesting observation. Could you please send a few examples of what LVG generates? Both sensical and non :)
>>>
>>> Dima
>>>
>>>
>>>
>>>
>>> On Apr 17, 2014, at 11:28, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>>>
>>>> The LVG annotator creates an enormous number of "lemmas" for every
>>>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>>>> think this is probably a minor bottleneck for speed but mostly a pretty
>>>> big space hog (at least 50% of the space of xmi files in my tests).
>>>>
>>>> As of right now I'm not sure if any downstream components are using
>>>> these lemmas, and on a manual inspection the precision seems to be
>>>> pretty abysmal (meaning most of them are nonsensical as lexical
>>>> variants), so as I said, just wondering if we can revisit why cTAKES
>>>> generates so many and whether that component can be optimized.
>>>>
>>>> Thanks
>>>> Tim
>>>>
>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Pei and I had a similar discussion in person -- mapping from lexical
variants to a stem might be useful. Pei also mentioned that one intended
use might have been searching the dictionary with lexical variants, but
I don't think that is done. Looking at the precision of the variants, I
think its highly unlikely the speed tradeoff would be worth any
improvements in recall.
Finally, at least in eclipse doing a search on references to the method
to retrieve the lemma entries turns up nothing.
Tim
On 04/17/2014 01:14 PM, Dligach, Dmitriy wrote:
> I don’t know of any applications within cTAKES that make use of this… The reverse (mapping from these “variants” to the normal form) may be useful though.
>
> Dima
>
>
>
>
> On Apr 17, 2014, at 11:50, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>
>> Sure, just as an example, I gave it a note with about 1000 words. It
>> generates 11500 NonEmptyFSList elements (each is basically one lexical
>> variant).
>>
>> For the word "symptomatic", these are the first 10 of 20 lexical variants:
>> Symptomaticer/JJ
>> Symptomaticer/RB
>> Symptomaticed/VB
>> Symptomaticcing/VB
>> Symptomatics/VB
>> Symptomatics/NN
>> Symptomaticked/VB
>> Symptomatic/VB
>> Symptomatic/JJ
>> Symptomatic/RB
>>
>> Tim
>>
>>
>> On 04/17/2014 12:31 PM, Dligach, Dmitriy wrote:
>>> Tim, this is a very interesting observation. Could you please send a few examples of what LVG generates? Both sensical and non :)
>>>
>>> Dima
>>>
>>>
>>>
>>>
>>> On Apr 17, 2014, at 11:28, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>>>
>>>> The LVG annotator creates an enormous number of "lemmas" for every
>>>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>>>> think this is probably a minor bottleneck for speed but mostly a pretty
>>>> big space hog (at least 50% of the space of xmi files in my tests).
>>>>
>>>> As of right now I'm not sure if any downstream components are using
>>>> these lemmas, and on a manual inspection the precision seems to be
>>>> pretty abysmal (meaning most of them are nonsensical as lexical
>>>> variants), so as I said, just wondering if we can revisit why cTAKES
>>>> generates so many and whether that component can be optimized.
>>>>
>>>> Thanks
>>>> Tim
>>>>
>
Re: lvg entries
Posted by "Dligach, Dmitriy" <Dm...@childrens.harvard.edu>.
I don’t know of any applications within cTAKES that make use of this… The reverse (mapping from these “variants” to the normal form) may be useful though.
Dima
On Apr 17, 2014, at 11:50, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
> Sure, just as an example, I gave it a note with about 1000 words. It
> generates 11500 NonEmptyFSList elements (each is basically one lexical
> variant).
>
> For the word "symptomatic", these are the first 10 of 20 lexical variants:
> Symptomaticer/JJ
> Symptomaticer/RB
> Symptomaticed/VB
> Symptomaticcing/VB
> Symptomatics/VB
> Symptomatics/NN
> Symptomaticked/VB
> Symptomatic/VB
> Symptomatic/JJ
> Symptomatic/RB
>
> Tim
>
>
> On 04/17/2014 12:31 PM, Dligach, Dmitriy wrote:
>> Tim, this is a very interesting observation. Could you please send a few examples of what LVG generates? Both sensical and non :)
>>
>> Dima
>>
>>
>>
>>
>> On Apr 17, 2014, at 11:28, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>>
>>> The LVG annotator creates an enormous number of "lemmas" for every
>>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>>> think this is probably a minor bottleneck for speed but mostly a pretty
>>> big space hog (at least 50% of the space of xmi files in my tests).
>>>
>>> As of right now I'm not sure if any downstream components are using
>>> these lemmas, and on a manual inspection the precision seems to be
>>> pretty abysmal (meaning most of them are nonsensical as lexical
>>> variants), so as I said, just wondering if we can revisit why cTAKES
>>> generates so many and whether that component can be optimized.
>>>
>>> Thanks
>>> Tim
>>>
>>
>
Re: lvg entries
Posted by "Miller, Timothy" <Ti...@childrens.harvard.edu>.
Sure, just as an example, I gave it a note with about 1000 words. It
generates 11500 NonEmptyFSList elements (each is basically one lexical
variant).
For the word "symptomatic", these are the first 10 of 20 lexical variants:
Symptomaticer/JJ
Symptomaticer/RB
Symptomaticed/VB
Symptomaticcing/VB
Symptomatics/VB
Symptomatics/NN
Symptomaticked/VB
Symptomatic/VB
Symptomatic/JJ
Symptomatic/RB
Tim
On 04/17/2014 12:31 PM, Dligach, Dmitriy wrote:
> Tim, this is a very interesting observation. Could you please send a few examples of what LVG generates? Both sensical and non :)
>
> Dima
>
>
>
>
> On Apr 17, 2014, at 11:28, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
>
>> The LVG annotator creates an enormous number of "lemmas" for every
>> WordToken in the CAS, and I'm wondering what the original purpose was? I
>> think this is probably a minor bottleneck for speed but mostly a pretty
>> big space hog (at least 50% of the space of xmi files in my tests).
>>
>> As of right now I'm not sure if any downstream components are using
>> these lemmas, and on a manual inspection the precision seems to be
>> pretty abysmal (meaning most of them are nonsensical as lexical
>> variants), so as I said, just wondering if we can revisit why cTAKES
>> generates so many and whether that component can be optimized.
>>
>> Thanks
>> Tim
>>
>
Re: lvg entries
Posted by "Dligach, Dmitriy" <Dm...@childrens.harvard.edu>.
Tim, this is a very interesting observation. Could you please send a few examples of what LVG generates? Both sensical and non :)
Dima
On Apr 17, 2014, at 11:28, Miller, Timothy <Ti...@childrens.harvard.edu> wrote:
> The LVG annotator creates an enormous number of "lemmas" for every
> WordToken in the CAS, and I'm wondering what the original purpose was? I
> think this is probably a minor bottleneck for speed but mostly a pretty
> big space hog (at least 50% of the space of xmi files in my tests).
>
> As of right now I'm not sure if any downstream components are using
> these lemmas, and on a manual inspection the precision seems to be
> pretty abysmal (meaning most of them are nonsensical as lexical
> variants), so as I said, just wondering if we can revisit why cTAKES
> generates so many and whether that component can be optimized.
>
> Thanks
> Tim
>