You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Marshall Schor (JIRA)" <ui...@incubator.apache.org> on 2008/06/17 22:13:45 UTC
[jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable,
token-based dictionary lookup UIMA component
[ https://issues.apache.org/jira/browse/UIMA-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605709#action_12605709 ]
Marshall Schor commented on UIMA-1033:
--------------------------------------
Software grant for this has been received and recorded.
> ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
> ----------------------------------------------------------------------------------
>
> Key: UIMA-1033
> URL: https://issues.apache.org/jira/browse/UIMA-1033
> Project: UIMA
> Issue Type: New Feature
> Components: Sandbox
> Environment: Java 5
> Reporter: Michael Tanenblatt
> Priority: Minor
> Attachments: conceptMapper.zip, conceptMapper.zip.md5
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> ConceptMapper is a token-based dictionary lookup UIMA component. It was
> designed specifically to allow any external tokenizer that is a UIMA
> component to be used to tokenize its dictionary. Using the same tokenizer
> on both the dictionary and for subsequent text processing prevents
> situations where a particular dictionary entry is not found, though it
> exists, because it was tokenized differently than the text being processed.
> ConceptMapper is highly configurable, in terms of:
> * the way dictionary entries are mapped to resultant annotations
> * the way input documents are processed
> * the availability of multiple lookup strategies
> * its various output options.
> Additionally, a set of post-processing filters are supplied, as well as an
> interface to easily create new filters. This allows for overgenerating
> results during the lookup phase, if so desired, then reducing the result
> set according to particular rules.
> More details:
> The structure of the dictionary itself is quite flexible. Entries can have
> any number of variants (synonyms), and arbitrary features can be associated
> with dictionary entries. Individual variants inherit features from parent
> token (i.e., the canonical from), but can override them or add additional
> features. In the following sample dictionary entry, there are 5 variants of
> the canonical form, and as described earlier, each inherits the SemClass
> and POS attributes from the canonical form, with the exception of the
> variant "mesenteric fibromatosis (c48.1)", which overrides the value of the
> SemClass attribute (this is somewhat of a contrived example, just to make
> that point):
> <token canonical="abdominal fibromatosis" SemClass="Diagnosis" POS="NN">
> <variant base="abdominal fibromatosis" />
> <variant base="abdominal desmoid" />
> <variant base="mesenteric fibromatosis (c48.1)"
> SemClass="Diagnosis-Site" />
> <variant base="mesenteric fibromatosis" />
> <variant base="retroperitoneal fibromatosis" />
> </token>
> Input tokens are processed one span at a time, where both the token and
> span (usually a sentence) annotation type are configurable. Additionally,
> the particular feature of the token annotation to use for lookups can be
> specified, otherwise its covered text is used. Other input configuration
> settings are whether to use case sensitive matching, an optional class name
> of a stemmer to apply to the tokens, and a list of stop words to to ignore
> during lookup. One additional input control mechanism is the ability to
> skip tokens during lookups based on particular feature values. In this way,
> it is easy to skip, for example, all tokens with particular part of speech
> tags, or with some previously computed semantic class.
> Output is in the form of new annotations, and the type of resulting
> annotations can be specified in a descriptor file. The mapping from
> dictionary entry attributes to the result annotation features can also be
> specified. Additionally, a string containing the matched text, a list of
> matched tokens, and the span enclosing the match can be specified to be set
> in the result annotations. It is also possible to indicate dictionary
> attributes to write back into each of the matched tokens.
> Dictionary lookup is controlled by three parameters in the descriptor, one
> of which allows for order-independent lookup (i.e., A B == B A), another
> togles between finding only the longest match vs. finding all possible
> matches. The final parameter specifies the search strategy, of which there
> are three. The default search strategy only considers contiguous tokens
> (not including tokens frm the stop word list or otherwise skipped tokens),
> and then begins the subsequent search after the longest match. The second
> strategy allows for ignoring non-matching tokens, allowing for disjoint
> matches, so that a dictionary entry of
> A C
> would match against the text
> A B C
> As with the default search strategy, the subsequent search begins after the
> longest match. The final search strategy is identical to the previous,
> except that subsequent searches begin one token ahead, instead of after the
> previous match. This enables overlapped matching.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Ahmed Abdeen Hamed <ah...@gmail.com>.
Hi Michael,Thank you for the follow up and the link. I decided to continue
to work with the ConceptMapper project since I have made good progress using
it. Please keep up posted should you have any documentation for it.
Best wishes,
Ahmed
On Thu, Jun 19, 2008 at 5:18 AM, Michael Baessler <mb...@michael-baessler.de>
wrote:
> Hi Ahmed,
>
> here is the link of the discussion.
>
> http://www.mail-archive.com/uima-user@incubator.apache.org/msg01277.html
>
> There are some minor differences in the capabilities. I think the
> discussion will show you the
> details and help you to make your decision which component do you need. For
> the DictionaryAnnotator
> there is a official release with documentation available. You get it with
> the Annotator-Addons
> package on the UIMA download page.
>
> -- Michael
>
> Michael Tanenblatt wrote:
> > There is some in-depth discussion about this in the UIMA User mailing
> > list--check the archives. The subject line was "Any interest in this as
> > an open source project?", and it was from May 2008 or possibly started
> > at the end of April.
> >
> >
> > On Jun 18, 2008, at 12:33 PM, Ahmed Abdeen Hamed wrote:
> >
> >> Thanks for the response. I am still not sure about some aspects of it. I
> >> just found out that the UIMA framework has this following
> >> DictionaryAnnotator feature:
> >>
> http://svn.apache.org/repos/asf/incubator/uima/sandbox/trunk/DictionaryAnnotator/doc/pdf/DictionaryAnnotatorUserGuide.pdf
> >>
> >>
> >> This is similar to what the ConceptMapper doing. Is there any
> >> advantage over
> >> the DictionaryAnnotator?
> >>
> >> Thank you!
> >> Ahmed
> >>
> >> On Wed, Jun 18, 2008 at 10:23 AM, Michael Tanenblatt <
> >> slothrop@park-slope.net> wrote:
> >>
> >>> My original message regarding this talks some about the dictionary
> >>> format.
> >>> I am in the process o writing a paper describing the whole of
> >>> ConceptMapper,
> >>> but that is not yet done. Here is what I wrote before:
> >>>
> >>> The structure of the dictionary itself is quite flexible. Entries can
> >>> have
> >>>> any number of variants (synonyms), and arbitrary features can be
> >>>> associated
> >>>> with dictionary entries. Individual variants inherit features from
> >>>> parent
> >>>> token (i.e., the canonical from), but can override them or add
> >>>> additional
> >>>> features. In the following sample dictionary entry, there are 5
> >>>> variants
> >>>> of
> >>>> the canonical form, and as described earlier, each inherits the
> >>>> SemClass
> >>>> and POS attributes from the canonical form, with the exception of the
> >>>> variant "mesenteric fibromatosis (c48.1)", which overrides the value
> of
> >>>> the
> >>>> SemClass attribute (this is somewhat of a contrived example, just to
> >>>> make
> >>>> that point):
> >>>> <token canonical="abdominal fibromatosis" SemClass="Diagnosis"
> >>>> POS="NN">
> >>>> <variant base="abdominal fibromatosis" />
> >>>> <variant base="abdominal desmoid" />
> >>>> <variant base="mesenteric fibromatosis (c48.1)"
> >>>> SemClass="Diagnosis-Site" />
> >>>> <variant base="mesenteric fibromatosis" />
> >>>> <variant base="retroperitoneal fibromatosis" />
> >>>> </token>
> >>>>
> >>>
> >>> So, testDict.xml is just an example. Two key AE descriptor parameters
> >>> are
> >>> "AttributeList" and "FeatureList", which provide the means to map
> >>> from the
> >>> XML attributes to the target annotation features. If your target
> >>> annotation
> >>> were called "DictTerm" and the DictTerm had the features
> >>> "canonicalForm",
> >>> "semanticClass" and "partOfSpeechTag", using the example dictionary
> >>> snippet
> >>> shown above, you would set AttributeList to:
> >>>
> >>> DictCanon
> >>> SemClass
> >>> POS
> >>>
> >>> and you would set FeatureList to:
> >>>
> >>> canonicalForm
> >>> semanticClass
> >>> partOfSpeechTag
> >>>
> >>> then, when one of the variants is matched in the text, a new DictTerm
> >>> would
> >>> be created with its semanticClass set to the value of the SemClass
> >>> attribute
> >>> and its partOfSpeechTag set to the value of the POS attribute.
> >>>
> >>> One important point: matches are only performed against the strings
> >>> listed
> >>> as attributes to the "variant" tag's "base" attribute. It is common
> >>> practice
> >>> to have something like the "token" element with something like a
> >>> canonical
> >>> form that is the same as one of the variants, but that is not required.
> >>>
> >>> I hope this helps!
> >>>
> >>>
> >>>
> >>> On Jun 18, 2008, at 10:06 AM, Ahmed Abdeen Hamed wrote:
> >>>
> >>> Thank Michael! I only recently joined the list so I missed the early
> >>>> posting. I like this example a lot. I was able to get it to run
> >>>> using the
> >>>> document analyzer from the uimaj-example. I have some questions
> though:
> >>>> Is the testDict.xml just an arbitrary xml file which means any
> >>>> well-formed
> >>>> xml file should work? How do I get my own xml dictionary files to work
> >>>> without transforming them into the xml format in your testDict.xml
> >>>> file?
> >>>> Is
> >>>> there documentation for this so that I can understand it on my own
> >>>> without
> >>>> bugging the entire list?Thanks!
> >>>> Ahmed
> >>>>
> >>>> On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <
> >>>> slothrop@park-slope.net>
> >>>> wrote:
> >>>>
> >>>> As Thilo mentioned in an email from May 19, 2008, I forgot to
> >>>> include the
> >>>>> source for uima.tt.TokenAnnotation, but otherwise the code should be
> >>>>> fine.
> >>>>>
> >>>>> Additionally, the problem you are seeing is with OffsetTokenizer,
> >>>>> which
> >>>>> is
> >>>>> just a sample tokenizer--if you have another, more robust
> >>>>> tokenizer, you
> >>>>> don't need this OffsetTokenizer.
> >>>>>
> >>>>>
> >>>>>
> >>>
> >
>
>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable,
token-based dictionary lookup UIMA component
Posted by Michael Baessler <mb...@michael-baessler.de>.
Hi Ahmed,
here is the link of the discussion.
http://www.mail-archive.com/uima-user@incubator.apache.org/msg01277.html
There are some minor differences in the capabilities. I think the discussion will show you the
details and help you to make your decision which component do you need. For the DictionaryAnnotator
there is a official release with documentation available. You get it with the Annotator-Addons
package on the UIMA download page.
-- Michael
Michael Tanenblatt wrote:
> There is some in-depth discussion about this in the UIMA User mailing
> list--check the archives. The subject line was "Any interest in this as
> an open source project?", and it was from May 2008 or possibly started
> at the end of April.
>
>
> On Jun 18, 2008, at 12:33 PM, Ahmed Abdeen Hamed wrote:
>
>> Thanks for the response. I am still not sure about some aspects of it. I
>> just found out that the UIMA framework has this following
>> DictionaryAnnotator feature:
>> http://svn.apache.org/repos/asf/incubator/uima/sandbox/trunk/DictionaryAnnotator/doc/pdf/DictionaryAnnotatorUserGuide.pdf
>>
>>
>> This is similar to what the ConceptMapper doing. Is there any
>> advantage over
>> the DictionaryAnnotator?
>>
>> Thank you!
>> Ahmed
>>
>> On Wed, Jun 18, 2008 at 10:23 AM, Michael Tanenblatt <
>> slothrop@park-slope.net> wrote:
>>
>>> My original message regarding this talks some about the dictionary
>>> format.
>>> I am in the process o writing a paper describing the whole of
>>> ConceptMapper,
>>> but that is not yet done. Here is what I wrote before:
>>>
>>> The structure of the dictionary itself is quite flexible. Entries can
>>> have
>>>> any number of variants (synonyms), and arbitrary features can be
>>>> associated
>>>> with dictionary entries. Individual variants inherit features from
>>>> parent
>>>> token (i.e., the canonical from), but can override them or add
>>>> additional
>>>> features. In the following sample dictionary entry, there are 5
>>>> variants
>>>> of
>>>> the canonical form, and as described earlier, each inherits the
>>>> SemClass
>>>> and POS attributes from the canonical form, with the exception of the
>>>> variant "mesenteric fibromatosis (c48.1)", which overrides the value of
>>>> the
>>>> SemClass attribute (this is somewhat of a contrived example, just to
>>>> make
>>>> that point):
>>>> <token canonical="abdominal fibromatosis" SemClass="Diagnosis"
>>>> POS="NN">
>>>> <variant base="abdominal fibromatosis" />
>>>> <variant base="abdominal desmoid" />
>>>> <variant base="mesenteric fibromatosis (c48.1)"
>>>> SemClass="Diagnosis-Site" />
>>>> <variant base="mesenteric fibromatosis" />
>>>> <variant base="retroperitoneal fibromatosis" />
>>>> </token>
>>>>
>>>
>>> So, testDict.xml is just an example. Two key AE descriptor parameters
>>> are
>>> "AttributeList" and "FeatureList", which provide the means to map
>>> from the
>>> XML attributes to the target annotation features. If your target
>>> annotation
>>> were called "DictTerm" and the DictTerm had the features
>>> "canonicalForm",
>>> "semanticClass" and "partOfSpeechTag", using the example dictionary
>>> snippet
>>> shown above, you would set AttributeList to:
>>>
>>> DictCanon
>>> SemClass
>>> POS
>>>
>>> and you would set FeatureList to:
>>>
>>> canonicalForm
>>> semanticClass
>>> partOfSpeechTag
>>>
>>> then, when one of the variants is matched in the text, a new DictTerm
>>> would
>>> be created with its semanticClass set to the value of the SemClass
>>> attribute
>>> and its partOfSpeechTag set to the value of the POS attribute.
>>>
>>> One important point: matches are only performed against the strings
>>> listed
>>> as attributes to the "variant" tag's "base" attribute. It is common
>>> practice
>>> to have something like the "token" element with something like a
>>> canonical
>>> form that is the same as one of the variants, but that is not required.
>>>
>>> I hope this helps!
>>>
>>>
>>>
>>> On Jun 18, 2008, at 10:06 AM, Ahmed Abdeen Hamed wrote:
>>>
>>> Thank Michael! I only recently joined the list so I missed the early
>>>> posting. I like this example a lot. I was able to get it to run
>>>> using the
>>>> document analyzer from the uimaj-example. I have some questions though:
>>>> Is the testDict.xml just an arbitrary xml file which means any
>>>> well-formed
>>>> xml file should work? How do I get my own xml dictionary files to work
>>>> without transforming them into the xml format in your testDict.xml
>>>> file?
>>>> Is
>>>> there documentation for this so that I can understand it on my own
>>>> without
>>>> bugging the entire list?Thanks!
>>>> Ahmed
>>>>
>>>> On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <
>>>> slothrop@park-slope.net>
>>>> wrote:
>>>>
>>>> As Thilo mentioned in an email from May 19, 2008, I forgot to
>>>> include the
>>>>> source for uima.tt.TokenAnnotation, but otherwise the code should be
>>>>> fine.
>>>>>
>>>>> Additionally, the problem you are seeing is with OffsetTokenizer,
>>>>> which
>>>>> is
>>>>> just a sample tokenizer--if you have another, more robust
>>>>> tokenizer, you
>>>>> don't need this OffsetTokenizer.
>>>>>
>>>>>
>>>>>
>>>
>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Michael Tanenblatt <sl...@park-slope.net>.
There is some in-depth discussion about this in the UIMA User mailing
list--check the archives. The subject line was "Any interest in this
as an open source project?", and it was from May 2008 or possibly
started at the end of April.
On Jun 18, 2008, at 12:33 PM, Ahmed Abdeen Hamed wrote:
> Thanks for the response. I am still not sure about some aspects of
> it. I
> just found out that the UIMA framework has this following
> DictionaryAnnotator feature:
> http://svn.apache.org/repos/asf/incubator/uima/sandbox/trunk/DictionaryAnnotator/doc/pdf/DictionaryAnnotatorUserGuide.pdf
>
> This is similar to what the ConceptMapper doing. Is there any
> advantage over
> the DictionaryAnnotator?
>
> Thank you!
> Ahmed
>
> On Wed, Jun 18, 2008 at 10:23 AM, Michael Tanenblatt <
> slothrop@park-slope.net> wrote:
>
>> My original message regarding this talks some about the dictionary
>> format.
>> I am in the process o writing a paper describing the whole of
>> ConceptMapper,
>> but that is not yet done. Here is what I wrote before:
>>
>> The structure of the dictionary itself is quite flexible. Entries
>> can have
>>> any number of variants (synonyms), and arbitrary features can be
>>> associated
>>> with dictionary entries. Individual variants inherit features from
>>> parent
>>> token (i.e., the canonical from), but can override them or add
>>> additional
>>> features. In the following sample dictionary entry, there are 5
>>> variants
>>> of
>>> the canonical form, and as described earlier, each inherits the
>>> SemClass
>>> and POS attributes from the canonical form, with the exception of
>>> the
>>> variant "mesenteric fibromatosis (c48.1)", which overrides the
>>> value of
>>> the
>>> SemClass attribute (this is somewhat of a contrived example, just
>>> to make
>>> that point):
>>> <token canonical="abdominal fibromatosis" SemClass="Diagnosis"
>>> POS="NN">
>>> <variant base="abdominal fibromatosis" />
>>> <variant base="abdominal desmoid" />
>>> <variant base="mesenteric fibromatosis (c48.1)"
>>> SemClass="Diagnosis-Site" />
>>> <variant base="mesenteric fibromatosis" />
>>> <variant base="retroperitoneal fibromatosis" />
>>> </token>
>>>
>>
>> So, testDict.xml is just an example. Two key AE descriptor
>> parameters are
>> "AttributeList" and "FeatureList", which provide the means to map
>> from the
>> XML attributes to the target annotation features. If your target
>> annotation
>> were called "DictTerm" and the DictTerm had the features
>> "canonicalForm",
>> "semanticClass" and "partOfSpeechTag", using the example dictionary
>> snippet
>> shown above, you would set AttributeList to:
>>
>> DictCanon
>> SemClass
>> POS
>>
>> and you would set FeatureList to:
>>
>> canonicalForm
>> semanticClass
>> partOfSpeechTag
>>
>> then, when one of the variants is matched in the text, a new
>> DictTerm would
>> be created with its semanticClass set to the value of the SemClass
>> attribute
>> and its partOfSpeechTag set to the value of the POS attribute.
>>
>> One important point: matches are only performed against the strings
>> listed
>> as attributes to the "variant" tag's "base" attribute. It is common
>> practice
>> to have something like the "token" element with something like a
>> canonical
>> form that is the same as one of the variants, but that is not
>> required.
>>
>> I hope this helps!
>>
>>
>>
>> On Jun 18, 2008, at 10:06 AM, Ahmed Abdeen Hamed wrote:
>>
>> Thank Michael! I only recently joined the list so I missed the early
>>> posting. I like this example a lot. I was able to get it to run
>>> using the
>>> document analyzer from the uimaj-example. I have some questions
>>> though:
>>> Is the testDict.xml just an arbitrary xml file which means any
>>> well-formed
>>> xml file should work? How do I get my own xml dictionary files to
>>> work
>>> without transforming them into the xml format in your testDict.xml
>>> file?
>>> Is
>>> there documentation for this so that I can understand it on my own
>>> without
>>> bugging the entire list?Thanks!
>>> Ahmed
>>>
>>> On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <
>>> slothrop@park-slope.net>
>>> wrote:
>>>
>>> As Thilo mentioned in an email from May 19, 2008, I forgot to
>>> include the
>>>> source for uima.tt.TokenAnnotation, but otherwise the code should
>>>> be
>>>> fine.
>>>>
>>>> Additionally, the problem you are seeing is with OffsetTokenizer,
>>>> which
>>>> is
>>>> just a sample tokenizer--if you have another, more robust
>>>> tokenizer, you
>>>> don't need this OffsetTokenizer.
>>>>
>>>>
>>>>
>>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Ahmed Abdeen Hamed <ah...@gmail.com>.
Thanks for the response. I am still not sure about some aspects of it. I
just found out that the UIMA framework has this following
DictionaryAnnotator feature:
http://svn.apache.org/repos/asf/incubator/uima/sandbox/trunk/DictionaryAnnotator/doc/pdf/DictionaryAnnotatorUserGuide.pdf
This is similar to what the ConceptMapper doing. Is there any advantage over
the DictionaryAnnotator?
Thank you!
Ahmed
On Wed, Jun 18, 2008 at 10:23 AM, Michael Tanenblatt <
slothrop@park-slope.net> wrote:
> My original message regarding this talks some about the dictionary format.
> I am in the process o writing a paper describing the whole of ConceptMapper,
> but that is not yet done. Here is what I wrote before:
>
> The structure of the dictionary itself is quite flexible. Entries can have
>> any number of variants (synonyms), and arbitrary features can be
>> associated
>> with dictionary entries. Individual variants inherit features from parent
>> token (i.e., the canonical from), but can override them or add additional
>> features. In the following sample dictionary entry, there are 5 variants
>> of
>> the canonical form, and as described earlier, each inherits the SemClass
>> and POS attributes from the canonical form, with the exception of the
>> variant "mesenteric fibromatosis (c48.1)", which overrides the value of
>> the
>> SemClass attribute (this is somewhat of a contrived example, just to make
>> that point):
>> <token canonical="abdominal fibromatosis" SemClass="Diagnosis" POS="NN">
>> <variant base="abdominal fibromatosis" />
>> <variant base="abdominal desmoid" />
>> <variant base="mesenteric fibromatosis (c48.1)"
>> SemClass="Diagnosis-Site" />
>> <variant base="mesenteric fibromatosis" />
>> <variant base="retroperitoneal fibromatosis" />
>> </token>
>>
>
> So, testDict.xml is just an example. Two key AE descriptor parameters are
> "AttributeList" and "FeatureList", which provide the means to map from the
> XML attributes to the target annotation features. If your target annotation
> were called "DictTerm" and the DictTerm had the features "canonicalForm",
> "semanticClass" and "partOfSpeechTag", using the example dictionary snippet
> shown above, you would set AttributeList to:
>
> DictCanon
> SemClass
> POS
>
> and you would set FeatureList to:
>
> canonicalForm
> semanticClass
> partOfSpeechTag
>
> then, when one of the variants is matched in the text, a new DictTerm would
> be created with its semanticClass set to the value of the SemClass attribute
> and its partOfSpeechTag set to the value of the POS attribute.
>
> One important point: matches are only performed against the strings listed
> as attributes to the "variant" tag's "base" attribute. It is common practice
> to have something like the "token" element with something like a canonical
> form that is the same as one of the variants, but that is not required.
>
> I hope this helps!
>
>
>
> On Jun 18, 2008, at 10:06 AM, Ahmed Abdeen Hamed wrote:
>
> Thank Michael! I only recently joined the list so I missed the early
>> posting. I like this example a lot. I was able to get it to run using the
>> document analyzer from the uimaj-example. I have some questions though:
>> Is the testDict.xml just an arbitrary xml file which means any well-formed
>> xml file should work? How do I get my own xml dictionary files to work
>> without transforming them into the xml format in your testDict.xml file?
>> Is
>> there documentation for this so that I can understand it on my own without
>> bugging the entire list?Thanks!
>> Ahmed
>>
>> On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <
>> slothrop@park-slope.net>
>> wrote:
>>
>> As Thilo mentioned in an email from May 19, 2008, I forgot to include the
>>> source for uima.tt.TokenAnnotation, but otherwise the code should be
>>> fine.
>>>
>>> Additionally, the problem you are seeing is with OffsetTokenizer, which
>>> is
>>> just a sample tokenizer--if you have another, more robust tokenizer, you
>>> don't need this OffsetTokenizer.
>>>
>>>
>>>
>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Michael Tanenblatt <sl...@park-slope.net>.
My original message regarding this talks some about the dictionary
format. I am in the process o writing a paper describing the whole of
ConceptMapper, but that is not yet done. Here is what I wrote before:
> The structure of the dictionary itself is quite flexible. Entries
> can have
> any number of variants (synonyms), and arbitrary features can be
> associated
> with dictionary entries. Individual variants inherit features from
> parent
> token (i.e., the canonical from), but can override them or add
> additional
> features. In the following sample dictionary entry, there are 5
> variants of
> the canonical form, and as described earlier, each inherits the
> SemClass
> and POS attributes from the canonical form, with the exception of the
> variant "mesenteric fibromatosis (c48.1)", which overrides the value
> of the
> SemClass attribute (this is somewhat of a contrived example, just to
> make
> that point):
> <token canonical="abdominal fibromatosis" SemClass="Diagnosis"
> POS="NN">
> <variant base="abdominal fibromatosis" />
> <variant base="abdominal desmoid" />
> <variant base="mesenteric fibromatosis (c48.1)"
> SemClass="Diagnosis-Site" />
> <variant base="mesenteric fibromatosis" />
> <variant base="retroperitoneal fibromatosis" />
> </token>
So, testDict.xml is just an example. Two key AE descriptor parameters
are "AttributeList" and "FeatureList", which provide the means to map
from the XML attributes to the target annotation features. If your
target annotation were called "DictTerm" and the DictTerm had the
features "canonicalForm", "semanticClass" and "partOfSpeechTag", using
the example dictionary snippet shown above, you would set
AttributeList to:
DictCanon
SemClass
POS
and you would set FeatureList to:
canonicalForm
semanticClass
partOfSpeechTag
then, when one of the variants is matched in the text, a new DictTerm
would be created with its semanticClass set to the value of the
SemClass attribute and its partOfSpeechTag set to the value of the POS
attribute.
One important point: matches are only performed against the strings
listed as attributes to the "variant" tag's "base" attribute. It is
common practice to have something like the "token" element with
something like a canonical form that is the same as one of the
variants, but that is not required.
I hope this helps!
On Jun 18, 2008, at 10:06 AM, Ahmed Abdeen Hamed wrote:
> Thank Michael! I only recently joined the list so I missed the early
> posting. I like this example a lot. I was able to get it to run
> using the
> document analyzer from the uimaj-example. I have some questions
> though:
> Is the testDict.xml just an arbitrary xml file which means any well-
> formed
> xml file should work? How do I get my own xml dictionary files to work
> without transforming them into the xml format in your testDict.xml
> file? Is
> there documentation for this so that I can understand it on my own
> without
> bugging the entire list?Thanks!
> Ahmed
>
> On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <slothrop@park-slope.net
> >
> wrote:
>
>> As Thilo mentioned in an email from May 19, 2008, I forgot to
>> include the
>> source for uima.tt.TokenAnnotation, but otherwise the code should
>> be fine.
>>
>> Additionally, the problem you are seeing is with OffsetTokenizer,
>> which is
>> just a sample tokenizer--if you have another, more robust
>> tokenizer, you
>> don't need this OffsetTokenizer.
>>
>>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Ahmed Abdeen Hamed <ah...@gmail.com>.
Thank Michael! I only recently joined the list so I missed the early
posting. I like this example a lot. I was able to get it to run using the
document analyzer from the uimaj-example. I have some questions though:
Is the testDict.xml just an arbitrary xml file which means any well-formed
xml file should work? How do I get my own xml dictionary files to work
without transforming them into the xml format in your testDict.xml file? Is
there documentation for this so that I can understand it on my own without
bugging the entire list?Thanks!
Ahmed
On Tue, Jun 17, 2008 at 8:05 PM, Michael Tanenblatt <sl...@park-slope.net>
wrote:
> As Thilo mentioned in an email from May 19, 2008, I forgot to include the
> source for uima.tt.TokenAnnotation, but otherwise the code should be fine.
>
> Additionally, the problem you are seeing is with OffsetTokenizer, which is
> just a sample tokenizer--if you have another, more robust tokenizer, you
> don't need this OffsetTokenizer.
>
>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Michael Tanenblatt <sl...@park-slope.net>.
As Thilo mentioned in an email from May 19, 2008, I forgot to include
the source for uima.tt.TokenAnnotation, but otherwise the code should
be fine.
Additionally, the problem you are seeing is with OffsetTokenizer,
which is just a sample tokenizer--if you have another, more robust
tokenizer, you don't need this OffsetTokenizer.
On Jun 17, 2008, at 6:23 PM, Ahmed Abdeen Hamed wrote:
> I think I found the problem. In the class
> .....support.tokenizer.OffsetTokenizer.java the following code need to
> replace the existing code:
>
> TokenAnnotation returnVal = new TokenAnnotation(jcas);
>
> // System.out.println("token = " + token.toString() + " fold =
> " +
>
> // foldCase(token.toString()));
>
> returnVal.setText(stem(foldCase(token.toString())));
>
> returnVal.setBegin(start);
>
> returnVal.setEnd(offset);
>
>
> return returnVal;
>
> Then you need to regenerate the TokenAnnotation TypeSystem classes.
>
> Can someone confirm the correctness of this?
>
>
> A quick question: what is the uima.tt package for? And, is there a
> reason
> for not giving it a name similar to the other packages?
>
>
> Thanks!
>
> Ahmed
>
>
>
>
> On Tue, Jun 17, 2008 at 4:49 PM, Ahmed Abdeen Hamed <ahmed.elmasri@gmail.com
> >
> wrote:
>
>> I happen to be in need of this feature when the email came out. I
>> downloaded the source and created an eclipse project for it.
>> However, the
>> project is not compiling. All the errors are in the ...../tokenizer
>> package.
>> I would appreciate getting this to compile.
>> Thanks,
>> Ahmed
>>
>>
>> On Tue, Jun 17, 2008 at 4:13 PM, Marshall Schor (JIRA) <
>> uima-dev@incubator.apache.org> wrote:
>>
>>>
>>> [
>>> https://issues.apache.org/jira/browse/UIMA-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605709
>>> #action_12605709]
>>>
>>> Marshall Schor commented on UIMA-1033:
>>> --------------------------------------
>>>
>>> Software grant for this has been received and recorded.
>>>
>>>> ConceptMapper--a highly configurable, token-based dictionary
>>>> lookup UIMA
>>> component
>>>>
>>> ----------------------------------------------------------------------------------
>>>>
>>>> Key: UIMA-1033
>>>> URL: https://issues.apache.org/jira/browse/UIMA-1033
>>>> Project: UIMA
>>>> Issue Type: New Feature
>>>> Components: Sandbox
>>>> Environment: Java 5
>>>> Reporter: Michael Tanenblatt
>>>> Priority: Minor
>>>> Attachments: conceptMapper.zip, conceptMapper.zip.md5
>>>>
>>>> Original Estimate: 24h
>>>> Remaining Estimate: 24h
>>>>
>>>> ConceptMapper is a token-based dictionary lookup UIMA component.
>>>> It was
>>>> designed specifically to allow any external tokenizer that is a
>>>> UIMA
>>>> component to be used to tokenize its dictionary. Using the same
>>> tokenizer
>>>> on both the dictionary and for subsequent text processing prevents
>>>> situations where a particular dictionary entry is not found,
>>>> though it
>>>> exists, because it was tokenized differently than the text being
>>> processed.
>>>> ConceptMapper is highly configurable, in terms of:
>>>> * the way dictionary entries are mapped to resultant annotations
>>>> * the way input documents are processed
>>>> * the availability of multiple lookup strategies
>>>> * its various output options.
>>>> Additionally, a set of post-processing filters are supplied, as
>>>> well as
>>> an
>>>> interface to easily create new filters. This allows for
>>>> overgenerating
>>>> results during the lookup phase, if so desired, then reducing the
>>>> result
>>>> set according to particular rules.
>>>> More details:
>>>> The structure of the dictionary itself is quite flexible. Entries
>>>> can
>>> have
>>>> any number of variants (synonyms), and arbitrary features can be
>>> associated
>>>> with dictionary entries. Individual variants inherit features from
>>> parent
>>>> token (i.e., the canonical from), but can override them or add
>>> additional
>>>> features. In the following sample dictionary entry, there are 5
>>>> variants
>>> of
>>>> the canonical form, and as described earlier, each inherits the
>>>> SemClass
>>>> and POS attributes from the canonical form, with the exception of
>>>> the
>>>> variant "mesenteric fibromatosis (c48.1)", which overrides the
>>>> value of
>>> the
>>>> SemClass attribute (this is somewhat of a contrived example, just
>>>> to
>>> make
>>>> that point):
>>>> <token canonical="abdominal fibromatosis" SemClass="Diagnosis"
>>>> POS="NN">
>>>> <variant base="abdominal fibromatosis" />
>>>> <variant base="abdominal desmoid" />
>>>> <variant base="mesenteric fibromatosis (c48.1)"
>>>> SemClass="Diagnosis-Site" />
>>>> <variant base="mesenteric fibromatosis" />
>>>> <variant base="retroperitoneal fibromatosis" />
>>>> </token>
>>>> Input tokens are processed one span at a time, where both the
>>>> token and
>>>> span (usually a sentence) annotation type are configurable.
>>> Additionally,
>>>> the particular feature of the token annotation to use for lookups
>>>> can be
>>>> specified, otherwise its covered text is used. Other input
>>>> configuration
>>>> settings are whether to use case sensitive matching, an optional
>>>> class
>>> name
>>>> of a stemmer to apply to the tokens, and a list of stop words to to
>>> ignore
>>>> during lookup. One additional input control mechanism is the
>>>> ability to
>>>> skip tokens during lookups based on particular feature values. In
>>>> this
>>> way,
>>>> it is easy to skip, for example, all tokens with particular part of
>>> speech
>>>> tags, or with some previously computed semantic class.
>>>> Output is in the form of new annotations, and the type of resulting
>>>> annotations can be specified in a descriptor file. The mapping from
>>>> dictionary entry attributes to the result annotation features can
>>>> also
>>> be
>>>> specified. Additionally, a string containing the matched text, a
>>>> list of
>>>> matched tokens, and the span enclosing the match can be specified
>>>> to be
>>> set
>>>> in the result annotations. It is also possible to indicate
>>>> dictionary
>>>> attributes to write back into each of the matched tokens.
>>>> Dictionary lookup is controlled by three parameters in the
>>>> descriptor,
>>> one
>>>> of which allows for order-independent lookup (i.e., A B == B A),
>>>> another
>>>> togles between finding only the longest match vs. finding all
>>>> possible
>>>> matches. The final parameter specifies the search strategy, of
>>>> which
>>> there
>>>> are three. The default search strategy only considers contiguous
>>>> tokens
>>>> (not including tokens frm the stop word list or otherwise skipped
>>> tokens),
>>>> and then begins the subsequent search after the longest match. The
>>> second
>>>> strategy allows for ignoring non-matching tokens, allowing for
>>>> disjoint
>>>> matches, so that a dictionary entry of
>>>> A C
>>>> would match against the text
>>>> A B C
>>>> As with the default search strategy, the subsequent search begins
>>>> after
>>> the
>>>> longest match. The final search strategy is identical to the
>>>> previous,
>>>> except that subsequent searches begin one token ahead, instead of
>>>> after
>>> the
>>>> previous match. This enables overlapped matching.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Ahmed Abdeen Hamed <ah...@gmail.com>.
I think I found the problem. In the class
.....support.tokenizer.OffsetTokenizer.java the following code need to
replace the existing code:
TokenAnnotation returnVal = new TokenAnnotation(jcas);
// System.out.println("token = " + token.toString() + " fold = " +
// foldCase(token.toString()));
returnVal.setText(stem(foldCase(token.toString())));
returnVal.setBegin(start);
returnVal.setEnd(offset);
return returnVal;
Then you need to regenerate the TokenAnnotation TypeSystem classes.
Can someone confirm the correctness of this?
A quick question: what is the uima.tt package for? And, is there a reason
for not giving it a name similar to the other packages?
Thanks!
Ahmed
On Tue, Jun 17, 2008 at 4:49 PM, Ahmed Abdeen Hamed <ah...@gmail.com>
wrote:
> I happen to be in need of this feature when the email came out. I
> downloaded the source and created an eclipse project for it. However, the
> project is not compiling. All the errors are in the ...../tokenizer package.
> I would appreciate getting this to compile.
> Thanks,
> Ahmed
>
>
> On Tue, Jun 17, 2008 at 4:13 PM, Marshall Schor (JIRA) <
> uima-dev@incubator.apache.org> wrote:
>
>>
>> [
>> https://issues.apache.org/jira/browse/UIMA-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605709#action_12605709]
>>
>> Marshall Schor commented on UIMA-1033:
>> --------------------------------------
>>
>> Software grant for this has been received and recorded.
>>
>> > ConceptMapper--a highly configurable, token-based dictionary lookup UIMA
>> component
>> >
>> ----------------------------------------------------------------------------------
>> >
>> > Key: UIMA-1033
>> > URL: https://issues.apache.org/jira/browse/UIMA-1033
>> > Project: UIMA
>> > Issue Type: New Feature
>> > Components: Sandbox
>> > Environment: Java 5
>> > Reporter: Michael Tanenblatt
>> > Priority: Minor
>> > Attachments: conceptMapper.zip, conceptMapper.zip.md5
>> >
>> > Original Estimate: 24h
>> > Remaining Estimate: 24h
>> >
>> > ConceptMapper is a token-based dictionary lookup UIMA component. It was
>> > designed specifically to allow any external tokenizer that is a UIMA
>> > component to be used to tokenize its dictionary. Using the same
>> tokenizer
>> > on both the dictionary and for subsequent text processing prevents
>> > situations where a particular dictionary entry is not found, though it
>> > exists, because it was tokenized differently than the text being
>> processed.
>> > ConceptMapper is highly configurable, in terms of:
>> > * the way dictionary entries are mapped to resultant annotations
>> > * the way input documents are processed
>> > * the availability of multiple lookup strategies
>> > * its various output options.
>> > Additionally, a set of post-processing filters are supplied, as well as
>> an
>> > interface to easily create new filters. This allows for overgenerating
>> > results during the lookup phase, if so desired, then reducing the result
>> > set according to particular rules.
>> > More details:
>> > The structure of the dictionary itself is quite flexible. Entries can
>> have
>> > any number of variants (synonyms), and arbitrary features can be
>> associated
>> > with dictionary entries. Individual variants inherit features from
>> parent
>> > token (i.e., the canonical from), but can override them or add
>> additional
>> > features. In the following sample dictionary entry, there are 5 variants
>> of
>> > the canonical form, and as described earlier, each inherits the SemClass
>> > and POS attributes from the canonical form, with the exception of the
>> > variant "mesenteric fibromatosis (c48.1)", which overrides the value of
>> the
>> > SemClass attribute (this is somewhat of a contrived example, just to
>> make
>> > that point):
>> > <token canonical="abdominal fibromatosis" SemClass="Diagnosis" POS="NN">
>> > <variant base="abdominal fibromatosis" />
>> > <variant base="abdominal desmoid" />
>> > <variant base="mesenteric fibromatosis (c48.1)"
>> > SemClass="Diagnosis-Site" />
>> > <variant base="mesenteric fibromatosis" />
>> > <variant base="retroperitoneal fibromatosis" />
>> > </token>
>> > Input tokens are processed one span at a time, where both the token and
>> > span (usually a sentence) annotation type are configurable.
>> Additionally,
>> > the particular feature of the token annotation to use for lookups can be
>> > specified, otherwise its covered text is used. Other input configuration
>> > settings are whether to use case sensitive matching, an optional class
>> name
>> > of a stemmer to apply to the tokens, and a list of stop words to to
>> ignore
>> > during lookup. One additional input control mechanism is the ability to
>> > skip tokens during lookups based on particular feature values. In this
>> way,
>> > it is easy to skip, for example, all tokens with particular part of
>> speech
>> > tags, or with some previously computed semantic class.
>> > Output is in the form of new annotations, and the type of resulting
>> > annotations can be specified in a descriptor file. The mapping from
>> > dictionary entry attributes to the result annotation features can also
>> be
>> > specified. Additionally, a string containing the matched text, a list of
>> > matched tokens, and the span enclosing the match can be specified to be
>> set
>> > in the result annotations. It is also possible to indicate dictionary
>> > attributes to write back into each of the matched tokens.
>> > Dictionary lookup is controlled by three parameters in the descriptor,
>> one
>> > of which allows for order-independent lookup (i.e., A B == B A), another
>> > togles between finding only the longest match vs. finding all possible
>> > matches. The final parameter specifies the search strategy, of which
>> there
>> > are three. The default search strategy only considers contiguous tokens
>> > (not including tokens frm the stop word list or otherwise skipped
>> tokens),
>> > and then begins the subsequent search after the longest match. The
>> second
>> > strategy allows for ignoring non-matching tokens, allowing for disjoint
>> > matches, so that a dictionary entry of
>> > A C
>> > would match against the text
>> > A B C
>> > As with the default search strategy, the subsequent search begins after
>> the
>> > longest match. The final search strategy is identical to the previous,
>> > except that subsequent searches begin one token ahead, instead of after
>> the
>> > previous match. This enables overlapped matching.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>
Re: [jira] Commented: (UIMA-1033) ConceptMapper--a highly configurable, token-based dictionary lookup UIMA component
Posted by Ahmed Abdeen Hamed <ah...@gmail.com>.
I happen to be in need of this feature when the email came out. I downloaded
the source and created an eclipse project for it. However, the project is
not compiling. All the errors are in the ...../tokenizer package.
I would appreciate getting this to compile.
Thanks,
Ahmed
On Tue, Jun 17, 2008 at 4:13 PM, Marshall Schor (JIRA) <
uima-dev@incubator.apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/UIMA-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605709#action_12605709]
>
> Marshall Schor commented on UIMA-1033:
> --------------------------------------
>
> Software grant for this has been received and recorded.
>
> > ConceptMapper--a highly configurable, token-based dictionary lookup UIMA
> component
> >
> ----------------------------------------------------------------------------------
> >
> > Key: UIMA-1033
> > URL: https://issues.apache.org/jira/browse/UIMA-1033
> > Project: UIMA
> > Issue Type: New Feature
> > Components: Sandbox
> > Environment: Java 5
> > Reporter: Michael Tanenblatt
> > Priority: Minor
> > Attachments: conceptMapper.zip, conceptMapper.zip.md5
> >
> > Original Estimate: 24h
> > Remaining Estimate: 24h
> >
> > ConceptMapper is a token-based dictionary lookup UIMA component. It was
> > designed specifically to allow any external tokenizer that is a UIMA
> > component to be used to tokenize its dictionary. Using the same tokenizer
> > on both the dictionary and for subsequent text processing prevents
> > situations where a particular dictionary entry is not found, though it
> > exists, because it was tokenized differently than the text being
> processed.
> > ConceptMapper is highly configurable, in terms of:
> > * the way dictionary entries are mapped to resultant annotations
> > * the way input documents are processed
> > * the availability of multiple lookup strategies
> > * its various output options.
> > Additionally, a set of post-processing filters are supplied, as well as
> an
> > interface to easily create new filters. This allows for overgenerating
> > results during the lookup phase, if so desired, then reducing the result
> > set according to particular rules.
> > More details:
> > The structure of the dictionary itself is quite flexible. Entries can
> have
> > any number of variants (synonyms), and arbitrary features can be
> associated
> > with dictionary entries. Individual variants inherit features from parent
> > token (i.e., the canonical from), but can override them or add additional
> > features. In the following sample dictionary entry, there are 5 variants
> of
> > the canonical form, and as described earlier, each inherits the SemClass
> > and POS attributes from the canonical form, with the exception of the
> > variant "mesenteric fibromatosis (c48.1)", which overrides the value of
> the
> > SemClass attribute (this is somewhat of a contrived example, just to make
> > that point):
> > <token canonical="abdominal fibromatosis" SemClass="Diagnosis" POS="NN">
> > <variant base="abdominal fibromatosis" />
> > <variant base="abdominal desmoid" />
> > <variant base="mesenteric fibromatosis (c48.1)"
> > SemClass="Diagnosis-Site" />
> > <variant base="mesenteric fibromatosis" />
> > <variant base="retroperitoneal fibromatosis" />
> > </token>
> > Input tokens are processed one span at a time, where both the token and
> > span (usually a sentence) annotation type are configurable. Additionally,
> > the particular feature of the token annotation to use for lookups can be
> > specified, otherwise its covered text is used. Other input configuration
> > settings are whether to use case sensitive matching, an optional class
> name
> > of a stemmer to apply to the tokens, and a list of stop words to to
> ignore
> > during lookup. One additional input control mechanism is the ability to
> > skip tokens during lookups based on particular feature values. In this
> way,
> > it is easy to skip, for example, all tokens with particular part of
> speech
> > tags, or with some previously computed semantic class.
> > Output is in the form of new annotations, and the type of resulting
> > annotations can be specified in a descriptor file. The mapping from
> > dictionary entry attributes to the result annotation features can also be
> > specified. Additionally, a string containing the matched text, a list of
> > matched tokens, and the span enclosing the match can be specified to be
> set
> > in the result annotations. It is also possible to indicate dictionary
> > attributes to write back into each of the matched tokens.
> > Dictionary lookup is controlled by three parameters in the descriptor,
> one
> > of which allows for order-independent lookup (i.e., A B == B A), another
> > togles between finding only the longest match vs. finding all possible
> > matches. The final parameter specifies the search strategy, of which
> there
> > are three. The default search strategy only considers contiguous tokens
> > (not including tokens frm the stop word list or otherwise skipped
> tokens),
> > and then begins the subsequent search after the longest match. The second
> > strategy allows for ignoring non-matching tokens, allowing for disjoint
> > matches, so that a dictionary entry of
> > A C
> > would match against the text
> > A B C
> > As with the default search strategy, the subsequent search begins after
> the
> > longest match. The final search strategy is identical to the previous,
> > except that subsequent searches begin one token ahead, instead of after
> the
> > previous match. This enables overlapped matching.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>