You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ctakes.apache.org by Tim O'Connell <ti...@gmail.com> on 2014/08/16 08:52:55 UTC
Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Hi folks,
I was having an issue with the current build (from svn) of ctakes/ytex not
identifying any annotations as some folks on this board. I traced it to
the fact that the UMLS database has at sometime in the relatively recent
past changed the SAB tag in the MRCONSO table for SNOMED terms from
SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
SNOMEDCT_US. Thus when the install script tried to create the
v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT terms,
thus nothing was getting annotated.
The ytex install script was just looking for things in MRCONSO with the
SNOMEDCT SAB tag when it created the ytex lookup table - so, by changing
this to SNOMEDCT_US in the file
CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
it now works (for mysql users) to find the annotations. You can just re-run
the ytex setup script, but that takes hours - instead, I just deleted all
the data from the v_snomed_fword_lookup table and basically ran the sql
command to repopulate the table and it worked fine. Here's the code, n.b.
my schema name for my umls database is 'umls' - change the code below if
yours is different.
delete from v_snomed_fword_lookup;
insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
stem_str)
select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
from umls_aui_fword c
inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
'SNOMEDCT_US', 'RXNORM')
inner join
(
select cui, min(tui) tui
from umls.MRSTY sty
where sty.tui in
(
'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
'T190', 'T191', 'T033',
'T184',
'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
'T018', 'T021', 'T024',
'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
'T110', 'T127',
'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
'T074', 'T075',
'T059'
)
group by cui
) t on t.cui = mrc.cui
;
Hope it helps - cheers,
Tim
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by Clayton Turner <ca...@g.cofc.edu>.
It didn't fix the @db.schema@ - I just went in and manually changed it
whenever the CPE complained. I assume that's supposed to be reading from
ytex.properties, but mine was set and it didn't resolve that @db.schema@
issue.
On Thu, Aug 21, 2014 at 5:00 PM, John Green <jo...@gmail.com>
wrote:
> Clayton - this indeed did fix the @db.schema@ for you? Im gonna try and
> reproduce (havent had time yet) then ill close the Jira ticket out.
>
>
> JG
> —
> Sent from Mailbox for iPhone
>
> On Thu, Aug 21, 2014 at 1:24 PM, Clayton Turner <ca...@g.cofc.edu>
> wrote:
>
> > Ah, I just switched to the ytex branch and all is good now. The SNOMED_US
> > issue has been plaguing me for weeks now so thanks a million for that.
> > On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner <ca...@g.cofc.edu>
> > wrote:
> >> Awesome. This is just what I needed for the longest time.
> >>
> >> I'm having a slight issue. When running either the ytex pipeline or ytex
> >> version of the AggregatePlaintextUMLSProcessor I get an error during
> >> initialization.
> >>
> >> My DictionaryLookupAnnotator.xml is raising a
> >> org.apache.uima.resource.ResourceInitializationException causedby:
> >> java.lang.ClassNotFoundException:
> >> edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
> >>
> >> I feel like I may have drifted away from what I need, though, because
> >> before this the CPE was complaining about a lack of
> LookupDesc_SNOMED.xml
> >> file. I found a ytex version of this on a google code site somewhere and
> >> pasted it where the CPE was looking for it. Now this error is coming up.
> >>
> >> Could my problem be solved with just a re-run of the ant script (was
> just
> >> trying to avoid since it takes ages) or is it a different issue?
> >>
> >>
> >> On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell <tim.oconnell@gmail.com
> >
> >> wrote:
> >>
> >>> Hi John,
> >>>
> >>> I'm not sure what was going on with the @db.schema@ error, although I
> was
> >>> getting it as well before with my prior build of 3.1.2 - I assume that
> >>> you've fixed something (thank you!) to make this go away. I rebuilt
> >>> everything from scratch and it's working now.
> >>>
> >>> I think one other thing I had to change was that after I had finished
> the
> >>> install/build, the cTakes version of LookupDesc_Db.xml doesn't work (in
> >>> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I had
> to
> >>> copy in an older version of the file from 3.1.1 to get the default
> cTakes
> >>> AggregatePlaintextUMLSProcessor pipeline working, although please
> >>> double-check that as my memory is a little foggy.
> >>>
> >>> But yes, here's what I have working since re-building:
> >>> 1. ytex-pipeline.xml
> >>> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
> >>> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with swapping
> >>> the
> >>> LookupDesc_Db.xml file as above)
> >>>
> >>> I've even made modifications to the ytex version of
> LookupDesc_SNOMED.xml
> >>> to get it tagging Disease Disorders, along with database modifications
> to
> >>> have it store these entries as well, which is working great.
> Literally,
> >>> everything is working perfectly now.
> >>>
> >>> Still so much for me to learn! Let me know if you need any more
> details.
> >>>
> >>> All the best,
> >>> Tim
> >>>
> >>>
> >>>
> >>> On Tue, Aug 19, 2014 at 4:31 AM, John Green <
> john.travis.green@gmail.com>
> >>> wrote:
> >>>
> >>> > I have not had time to implement this - to clarify out of curiosity,
> >>> does
> >>> > this clear up the @db.schema@ error Tim? And did you successfully
> run
> >>> > ytex with the ctakes dictionary-lookup?
> >>> >
> >>> >
> >>> > JG
> >>> > —
> >>> > Sent from Mailbox for iPhone
> >>> >
> >>> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <
> tim.oconnell@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi folks,
> >>> > > I was having an issue with the current build (from svn) of
> ctakes/ytex
> >>> > not
> >>> > > identifying any annotations as some folks on this board. I traced
> it
> >>> to
> >>> > > the fact that the UMLS database has at sometime in the relatively
> >>> recent
> >>> > > past changed the SAB tag in the MRCONSO table for SNOMED terms from
> >>> > > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that
> uses
> >>> > > SNOMEDCT_US. Thus when the install script tried to create the
> >>> > > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT
> >>> terms,
> >>> > > thus nothing was getting annotated.
> >>> > > The ytex install script was just looking for things in MRCONSO with
> >>> the
> >>> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
> >>> changing
> >>> > > this to SNOMEDCT_US in the file
> >>> > >
> >>> >
> >>>
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> >>> > > it now works (for mysql users) to find the annotations. You can
> just
> >>> > re-run
> >>> > > the ytex setup script, but that takes hours - instead, I just
> deleted
> >>> all
> >>> > > the data from the v_snomed_fword_lookup table and basically ran the
> >>> sql
> >>> > > command to repopulate the table and it worked fine. Here's the
> code,
> >>> n.b.
> >>> > > my schema name for my umls database is 'umls' - change the code
> below
> >>> if
> >>> > > yours is different.
> >>> > > delete from v_snomed_fword_lookup;
> >>> > > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> >>> > > stem_str)
> >>> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> >>> > > from umls_aui_fword c
> >>> > > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> >>> > > 'SNOMEDCT_US', 'RXNORM')
> >>> > > inner join
> >>> > > (
> >>> > > select cui, min(tui) tui
> >>> > > from umls.MRSTY sty
> >>> > > where sty.tui in
> >>> > > (
> >>> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> >>> > > 'T190', 'T191', 'T033',
> >>> > > 'T184',
> >>> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> >>> > > 'T018', 'T021', 'T024',
> >>> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> >>> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129',
> 'T130',
> >>> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121',
> 'T192',
> >>> > > 'T110', 'T127',
> >>> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> >>> > > 'T074', 'T075',
> >>> > > 'T059'
> >>> > > )
> >>> > > group by cui
> >>> > > ) t on t.cui = mrc.cui
> >>> > > ;
> >>> > > Hope it helps - cheers,
> >>> > > Tim
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> --
> >> Clayton Turner
> >> email: caturner3@g.cofc.edu
> >> phone: (843)-424-3784
> >> web: claytonturner.blogspot.com
> >>
> >>
> -------------------------------------------------------------------------------------------------
> >> “When scientifically investigating the natural world, the only thing
> worse
> >> than a blind believer is a seeing denier.”
> >> - Neil deGrasse Tyson
> >>
> > --
> > --
> > Clayton Turner
> > email: caturner3@g.cofc.edu
> > phone: (843)-424-3784
> > web: claytonturner.blogspot.com
> >
> -------------------------------------------------------------------------------------------------
> > “When scientifically investigating the natural world, the only thing
> worse
> > than a blind believer is a seeing denier.”
> > - Neil deGrasse Tyson
>
--
--
Clayton Turner
email: caturner3@g.cofc.edu
phone: (843)-424-3784
web: claytonturner.blogspot.com
-------------------------------------------------------------------------------------------------
“When scientifically investigating the natural world, the only thing worse
than a blind believer is a seeing denier.”
- Neil deGrasse Tyson
Re: Change from SNOMEDCT to SNOMEDCT_US affecting
v_snomed_fword_lookup
Posted by John Green <jo...@gmail.com>.
Clayton - this indeed did fix the @db.schema@ for you? Im gonna try and reproduce (havent had time yet) then ill close the Jira ticket out.
JG
—
Sent from Mailbox for iPhone
On Thu, Aug 21, 2014 at 1:24 PM, Clayton Turner <ca...@g.cofc.edu>
wrote:
> Ah, I just switched to the ytex branch and all is good now. The SNOMED_US
> issue has been plaguing me for weeks now so thanks a million for that.
> On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner <ca...@g.cofc.edu>
> wrote:
>> Awesome. This is just what I needed for the longest time.
>>
>> I'm having a slight issue. When running either the ytex pipeline or ytex
>> version of the AggregatePlaintextUMLSProcessor I get an error during
>> initialization.
>>
>> My DictionaryLookupAnnotator.xml is raising a
>> org.apache.uima.resource.ResourceInitializationException causedby:
>> java.lang.ClassNotFoundException:
>> edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
>>
>> I feel like I may have drifted away from what I need, though, because
>> before this the CPE was complaining about a lack of LookupDesc_SNOMED.xml
>> file. I found a ytex version of this on a google code site somewhere and
>> pasted it where the CPE was looking for it. Now this error is coming up.
>>
>> Could my problem be solved with just a re-run of the ant script (was just
>> trying to avoid since it takes ages) or is it a different issue?
>>
>>
>> On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell <ti...@gmail.com>
>> wrote:
>>
>>> Hi John,
>>>
>>> I'm not sure what was going on with the @db.schema@ error, although I was
>>> getting it as well before with my prior build of 3.1.2 - I assume that
>>> you've fixed something (thank you!) to make this go away. I rebuilt
>>> everything from scratch and it's working now.
>>>
>>> I think one other thing I had to change was that after I had finished the
>>> install/build, the cTakes version of LookupDesc_Db.xml doesn't work (in
>>> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I had to
>>> copy in an older version of the file from 3.1.1 to get the default cTakes
>>> AggregatePlaintextUMLSProcessor pipeline working, although please
>>> double-check that as my memory is a little foggy.
>>>
>>> But yes, here's what I have working since re-building:
>>> 1. ytex-pipeline.xml
>>> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
>>> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with swapping
>>> the
>>> LookupDesc_Db.xml file as above)
>>>
>>> I've even made modifications to the ytex version of LookupDesc_SNOMED.xml
>>> to get it tagging Disease Disorders, along with database modifications to
>>> have it store these entries as well, which is working great. Literally,
>>> everything is working perfectly now.
>>>
>>> Still so much for me to learn! Let me know if you need any more details.
>>>
>>> All the best,
>>> Tim
>>>
>>>
>>>
>>> On Tue, Aug 19, 2014 at 4:31 AM, John Green <jo...@gmail.com>
>>> wrote:
>>>
>>> > I have not had time to implement this - to clarify out of curiosity,
>>> does
>>> > this clear up the @db.schema@ error Tim? And did you successfully run
>>> > ytex with the ctakes dictionary-lookup?
>>> >
>>> >
>>> > JG
>>> > —
>>> > Sent from Mailbox for iPhone
>>> >
>>> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi folks,
>>> > > I was having an issue with the current build (from svn) of ctakes/ytex
>>> > not
>>> > > identifying any annotations as some folks on this board. I traced it
>>> to
>>> > > the fact that the UMLS database has at sometime in the relatively
>>> recent
>>> > > past changed the SAB tag in the MRCONSO table for SNOMED terms from
>>> > > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
>>> > > SNOMEDCT_US. Thus when the install script tried to create the
>>> > > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT
>>> terms,
>>> > > thus nothing was getting annotated.
>>> > > The ytex install script was just looking for things in MRCONSO with
>>> the
>>> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
>>> changing
>>> > > this to SNOMEDCT_US in the file
>>> > >
>>> >
>>> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
>>> > > it now works (for mysql users) to find the annotations. You can just
>>> > re-run
>>> > > the ytex setup script, but that takes hours - instead, I just deleted
>>> all
>>> > > the data from the v_snomed_fword_lookup table and basically ran the
>>> sql
>>> > > command to repopulate the table and it worked fine. Here's the code,
>>> n.b.
>>> > > my schema name for my umls database is 'umls' - change the code below
>>> if
>>> > > yours is different.
>>> > > delete from v_snomed_fword_lookup;
>>> > > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
>>> > > stem_str)
>>> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
>>> > > from umls_aui_fword c
>>> > > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
>>> > > 'SNOMEDCT_US', 'RXNORM')
>>> > > inner join
>>> > > (
>>> > > select cui, min(tui) tui
>>> > > from umls.MRSTY sty
>>> > > where sty.tui in
>>> > > (
>>> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
>>> > > 'T190', 'T191', 'T033',
>>> > > 'T184',
>>> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
>>> > > 'T018', 'T021', 'T024',
>>> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
>>> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
>>> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
>>> > > 'T110', 'T127',
>>> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
>>> > > 'T074', 'T075',
>>> > > 'T059'
>>> > > )
>>> > > group by cui
>>> > > ) t on t.cui = mrc.cui
>>> > > ;
>>> > > Hope it helps - cheers,
>>> > > Tim
>>> >
>>>
>>
>>
>>
>> --
>> --
>> Clayton Turner
>> email: caturner3@g.cofc.edu
>> phone: (843)-424-3784
>> web: claytonturner.blogspot.com
>>
>> -------------------------------------------------------------------------------------------------
>> “When scientifically investigating the natural world, the only thing worse
>> than a blind believer is a seeing denier.”
>> - Neil deGrasse Tyson
>>
> --
> --
> Clayton Turner
> email: caturner3@g.cofc.edu
> phone: (843)-424-3784
> web: claytonturner.blogspot.com
> -------------------------------------------------------------------------------------------------
> “When scientifically investigating the natural world, the only thing worse
> than a blind believer is a seeing denier.”
> - Neil deGrasse Tyson
RE: Change from SNOMEDCT to SNOMEDCT_US affecting
v_snomed_fword_lookup
Posted by "Chen, Pei" <Pe...@childrens.harvard.edu>.
This has been done in trunk for the next patch 3.2.1 release:
https://issues.apache.org/jira/browse/CTAKES-309
Thanks for pointing this out- Would be great if someone could confirm the ytex scripts works as expected with their latest version of umls.
> -----Original Message-----
> From: vijay garla [mailto:vngarla@gmail.com]
> Sent: Thursday, August 21, 2014 11:07 PM
> To: dev@ctakes.apache.org
> Subject: Re: Change from SNOMEDCT to SNOMEDCT_US affecting
> v_snomed_fword_lookup
>
> That would definitely make sense
>
> On Thursday, August 21, 2014, Chen, Pei <Pe...@childrens.harvard.edu>
> wrote:
>
> > VJ,
> > Would it make sense to add in('SNOMEDCT_US') to the default
> > ctakes-ytex/scripts/data/**/insert_view.sql?
> > That way it'll support umls2011 as well as the newer 2014 naming
> > conventions?
> > Ex: inner join MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> > 'SNOMEDCT',
> > 'SNOMEDCT_US','RXNORM')
> >
> > --Pei
> >
> > > -----Original Message-----
> > > From: clayclay911@gmail.com <javascript:;>
> > > [mailto:clayclay911@gmail.com
> > <javascript:;>] On Behalf Of
> > > Clayton Turner
> > > Sent: Thursday, August 21, 2014 4:25 PM
> > > To: dev@ctakes.apache.org <javascript:;>
> > > Subject: Re: Change from SNOMEDCT to SNOMEDCT_US affecting
> > > v_snomed_fword_lookup
> > >
> > > Ah, I just switched to the ytex branch and all is good now. The
> > > SNOMED_US issue has been plaguing me for weeks now so thanks a
> million for that.
> > >
> > >
> > > On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner
> > > <caturner3@g.cofc.edu
> > <javascript:;>>
> > > wrote:
> > >
> > > > Awesome. This is just what I needed for the longest time.
> > > >
> > > > I'm having a slight issue. When running either the ytex pipeline
> > > > or ytex version of the AggregatePlaintextUMLSProcessor I get an
> > > > error during initialization.
> > > >
> > > > My DictionaryLookupAnnotator.xml is raising a
> > > > org.apache.uima.resource.ResourceInitializationException causedby:
> > > > java.lang.ClassNotFoundException:
> > > > edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
> > > >
> > > > I feel like I may have drifted away from what I need, though,
> > > > because before this the CPE was complaining about a lack of
> > > > LookupDesc_SNOMED.xml file. I found a ytex version of this on a
> > > > google code site somewhere and pasted it where the CPE was looking
> > > > for it. Now
> > > this error is coming up.
> > > >
> > > > Could my problem be solved with just a re-run of the ant script
> > > > (was just trying to avoid since it takes ages) or is it a different issue?
> > > >
> > > >
> > > > On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell
> > > > <tim.oconnell@gmail.com <javascript:;>>
> > > > wrote:
> > > >
> > > >> Hi John,
> > > >>
> > > >> I'm not sure what was going on with the @db.schema@ error,
> > > >> although I was getting it as well before with my prior build of
> > > >> 3.1.2 - I assume that you've fixed something (thank you!) to make
> > > >> this go away. I rebuilt everything from scratch and it's working now.
> > > >>
> > > >> I think one other thing I had to change was that after I had
> > > >> finished the install/build, the cTakes version of
> > > >> LookupDesc_Db.xml doesn't work (in
> > > >> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure
> > > >> I had to copy in an older version of the file from 3.1.1 to get
> > > >> the default cTakes AggregatePlaintextUMLSProcessor pipeline
> > > >> working, although please double-check that as my memory is a little
> foggy.
> > > >>
> > > >> But yes, here's what I have working since re-building:
> > > >> 1. ytex-pipeline.xml
> > > >> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
> > > >> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with
> > > >> swapping the LookupDesc_Db.xml file as above)
> > > >>
> > > >> I've even made modifications to the ytex version of
> > > >> LookupDesc_SNOMED.xml to get it tagging Disease Disorders, along
> > > >> with
> > > database modifications to
> > > >> have it store these entries as well, which is working great.
> > Literally,
> > > >> everything is working perfectly now.
> > > >>
> > > >> Still so much for me to learn! Let me know if you need any more
> > details.
> > > >>
> > > >> All the best,
> > > >> Tim
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Aug 19, 2014 at 4:31 AM, John Green
> > > >> <john.travis.green@gmail.com <javascript:;>>
> > > >> wrote:
> > > >>
> > > >> > I have not had time to implement this - to clarify out of
> > > >> > curiosity,
> > > >> does
> > > >> > this clear up the @db.schema@ error Tim? And did you
> > > >> > successfully run ytex with the ctakes dictionary-lookup?
> > > >> >
> > > >> >
> > > >> > JG
> > > >> > —
> > > >> > Sent from Mailbox for iPhone
> > > >> >
> > > >> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell
> > > >> > <tim.oconnell@gmail.com <javascript:;>>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi folks,
> > > >> > > I was having an issue with the current build (from svn) of
> > > >> > > ctakes/ytex
> > > >> > not
> > > >> > > identifying any annotations as some folks on this board. I
> > > >> > > traced it
> > > >> to
> > > >> > > the fact that the UMLS database has at sometime in the
> > > >> > > relatively
> > > >> recent
> > > >> > > past changed the SAB tag in the MRCONSO table for SNOMED
> > > >> > > terms from SNOMEDCT to SNOMEDCT_US. I just had a newer
> > > >> > > version of
> > > UMLS
> > > >> > > that uses SNOMEDCT_US. Thus when the install script tried to
> > > >> > > create the v_snomed_fword_lookup table, it wasn't finding any
> > > >> > > of the SNOMEDCT
> > > >> terms,
> > > >> > > thus nothing was getting annotated.
> > > >> > > The ytex install script was just looking for things in
> > > >> > > MRCONSO with
> > > >> the
> > > >> > > SNOMEDCT SAB tag when it created the ytex lookup table - so,
> > > >> > > by
> > > >> changing
> > > >> > > this to SNOMEDCT_US in the file
> > > >> > >
> > > >> >
> > > >> CTAKES_HOME/bin/ctakes-
> > > ytex/scripts/data/mysql/umls/insert_view_templ
> > > >> ate.sql
> > > >> > > it now works (for mysql users) to find the annotations. You
> > > >> > > can just
> > > >> > re-run
> > > >> > > the ytex setup script, but that takes hours - instead, I just
> > > >> > > deleted
> > > >> all
> > > >> > > the data from the v_snomed_fword_lookup table and basically
> > > >> > > ran the
> > > >> sql
> > > >> > > command to repopulate the table and it worked fine. Here's
> > > >> > > the code,
> > > >> n.b.
> > > >> > > my schema name for my umls database is 'umls' - change the
> > > >> > > code below
> > > >> if
> > > >> > > yours is different.
> > > >> > > delete from v_snomed_fword_lookup; insert into
> > > >> > > v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> > > >> > > stem_str)
> > > >> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str,
> > > >> > > c.stem_str from umls_aui_fword c inner join umls.MRCONSO mrc
> > > >> > > on c.aui = mrc.aui and mrc.SAB in ( 'SNOMEDCT_US', 'RXNORM')
> > > >> > > inner join ( select cui, min(tui) tui from umls.MRSTY sty where
> sty.tui in (
> > > >> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> > > >> > > 'T190', 'T191', 'T033',
> > > >> > > 'T184',
> > > >> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> > > >> > > 'T018', 'T021', 'T024',
> > > >> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120',
> > 'T104',
> > > >> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125',
> > > >> > > 'T129',
> > 'T130',
> > > >> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115',
> > > >> > > 'T121',
> > 'T192',
> > > >> > > 'T110', 'T127',
> > > >> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> > > >> > > 'T074', 'T075',
> > > >> > > 'T059'
> > > >> > > )
> > > >> > > group by cui
> > > >> > > ) t on t.cui = mrc.cui
> > > >> > > ;
> > > >> > > Hope it helps - cheers,
> > > >> > > Tim
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > Clayton Turner
> > > > email: caturner3@g.cofc.edu <javascript:;>
> > > > phone: (843)-424-3784
> > > > web: claytonturner.blogspot.com
> > > >
> > > > ------------------------------------------------------------------
> > > > ----
> > > > --------------------------- “When scientifically investigating the
> > > > natural world, the only thing worse than a blind believer is a
> > > > seeing denier.”
> > > > - Neil deGrasse Tyson
> > > >
> > >
> > >
> > >
> > > --
> > > --
> > > Clayton Turner
> > > email: caturner3@g.cofc.edu <javascript:;>
> > > phone: (843)-424-3784
> > > web: claytonturner.blogspot.com
> > >
> > ----------------------------------------------------------------------
> > ------------------------
> > > ---
> > > “When scientifically investigating the natural world, the only thing
> > worse than
> > > a blind believer is a seeing denier.”
> > > - Neil deGrasse Tyson
> >
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by vijay garla <vn...@gmail.com>.
That would definitely make sense
On Thursday, August 21, 2014, Chen, Pei <Pe...@childrens.harvard.edu>
wrote:
> VJ,
> Would it make sense to add in('SNOMEDCT_US') to the default
> ctakes-ytex/scripts/data/**/insert_view.sql?
> That way it'll support umls2011 as well as the newer 2014 naming
> conventions?
> Ex: inner join MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in ( 'SNOMEDCT',
> 'SNOMEDCT_US','RXNORM')
>
> --Pei
>
> > -----Original Message-----
> > From: clayclay911@gmail.com <javascript:;> [mailto:clayclay911@gmail.com
> <javascript:;>] On Behalf Of
> > Clayton Turner
> > Sent: Thursday, August 21, 2014 4:25 PM
> > To: dev@ctakes.apache.org <javascript:;>
> > Subject: Re: Change from SNOMEDCT to SNOMEDCT_US affecting
> > v_snomed_fword_lookup
> >
> > Ah, I just switched to the ytex branch and all is good now. The SNOMED_US
> > issue has been plaguing me for weeks now so thanks a million for that.
> >
> >
> > On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner <caturner3@g.cofc.edu
> <javascript:;>>
> > wrote:
> >
> > > Awesome. This is just what I needed for the longest time.
> > >
> > > I'm having a slight issue. When running either the ytex pipeline or
> > > ytex version of the AggregatePlaintextUMLSProcessor I get an error
> > > during initialization.
> > >
> > > My DictionaryLookupAnnotator.xml is raising a
> > > org.apache.uima.resource.ResourceInitializationException causedby:
> > > java.lang.ClassNotFoundException:
> > > edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
> > >
> > > I feel like I may have drifted away from what I need, though, because
> > > before this the CPE was complaining about a lack of
> > > LookupDesc_SNOMED.xml file. I found a ytex version of this on a google
> > > code site somewhere and pasted it where the CPE was looking for it. Now
> > this error is coming up.
> > >
> > > Could my problem be solved with just a re-run of the ant script (was
> > > just trying to avoid since it takes ages) or is it a different issue?
> > >
> > >
> > > On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell
> > > <tim.oconnell@gmail.com <javascript:;>>
> > > wrote:
> > >
> > >> Hi John,
> > >>
> > >> I'm not sure what was going on with the @db.schema@ error, although I
> > >> was getting it as well before with my prior build of 3.1.2 - I assume
> > >> that you've fixed something (thank you!) to make this go away. I
> > >> rebuilt everything from scratch and it's working now.
> > >>
> > >> I think one other thing I had to change was that after I had finished
> > >> the install/build, the cTakes version of LookupDesc_Db.xml doesn't
> > >> work (in
> > >> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I
> > >> had to copy in an older version of the file from 3.1.1 to get the
> > >> default cTakes AggregatePlaintextUMLSProcessor pipeline working,
> > >> although please double-check that as my memory is a little foggy.
> > >>
> > >> But yes, here's what I have working since re-building:
> > >> 1. ytex-pipeline.xml
> > >> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
> > >> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with
> > >> swapping the LookupDesc_Db.xml file as above)
> > >>
> > >> I've even made modifications to the ytex version of
> > >> LookupDesc_SNOMED.xml to get it tagging Disease Disorders, along with
> > database modifications to
> > >> have it store these entries as well, which is working great.
> Literally,
> > >> everything is working perfectly now.
> > >>
> > >> Still so much for me to learn! Let me know if you need any more
> details.
> > >>
> > >> All the best,
> > >> Tim
> > >>
> > >>
> > >>
> > >> On Tue, Aug 19, 2014 at 4:31 AM, John Green
> > >> <john.travis.green@gmail.com <javascript:;>>
> > >> wrote:
> > >>
> > >> > I have not had time to implement this - to clarify out of
> > >> > curiosity,
> > >> does
> > >> > this clear up the @db.schema@ error Tim? And did you successfully
> > >> > run ytex with the ctakes dictionary-lookup?
> > >> >
> > >> >
> > >> > JG
> > >> > —
> > >> > Sent from Mailbox for iPhone
> > >> >
> > >> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell
> > >> > <tim.oconnell@gmail.com <javascript:;>>
> > >> > wrote:
> > >> >
> > >> > > Hi folks,
> > >> > > I was having an issue with the current build (from svn) of
> > >> > > ctakes/ytex
> > >> > not
> > >> > > identifying any annotations as some folks on this board. I
> > >> > > traced it
> > >> to
> > >> > > the fact that the UMLS database has at sometime in the relatively
> > >> recent
> > >> > > past changed the SAB tag in the MRCONSO table for SNOMED terms
> > >> > > from SNOMEDCT to SNOMEDCT_US. I just had a newer version of
> > UMLS
> > >> > > that uses SNOMEDCT_US. Thus when the install script tried to
> > >> > > create the v_snomed_fword_lookup table, it wasn't finding any of
> > >> > > the SNOMEDCT
> > >> terms,
> > >> > > thus nothing was getting annotated.
> > >> > > The ytex install script was just looking for things in MRCONSO
> > >> > > with
> > >> the
> > >> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
> > >> changing
> > >> > > this to SNOMEDCT_US in the file
> > >> > >
> > >> >
> > >> CTAKES_HOME/bin/ctakes-
> > ytex/scripts/data/mysql/umls/insert_view_templ
> > >> ate.sql
> > >> > > it now works (for mysql users) to find the annotations. You can
> > >> > > just
> > >> > re-run
> > >> > > the ytex setup script, but that takes hours - instead, I just
> > >> > > deleted
> > >> all
> > >> > > the data from the v_snomed_fword_lookup table and basically ran
> > >> > > the
> > >> sql
> > >> > > command to repopulate the table and it worked fine. Here's the
> > >> > > code,
> > >> n.b.
> > >> > > my schema name for my umls database is 'umls' - change the code
> > >> > > below
> > >> if
> > >> > > yours is different.
> > >> > > delete from v_snomed_fword_lookup; insert into
> > >> > > v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> > >> > > stem_str)
> > >> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> > >> > > from umls_aui_fword c inner join umls.MRCONSO mrc on c.aui =
> > >> > > mrc.aui and mrc.SAB in ( 'SNOMEDCT_US', 'RXNORM') inner join (
> > >> > > select cui, min(tui) tui from umls.MRSTY sty where sty.tui in (
> > >> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> > >> > > 'T190', 'T191', 'T033',
> > >> > > 'T184',
> > >> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> > >> > > 'T018', 'T021', 'T024',
> > >> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120',
> 'T104',
> > >> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129',
> 'T130',
> > >> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121',
> 'T192',
> > >> > > 'T110', 'T127',
> > >> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> > >> > > 'T074', 'T075',
> > >> > > 'T059'
> > >> > > )
> > >> > > group by cui
> > >> > > ) t on t.cui = mrc.cui
> > >> > > ;
> > >> > > Hope it helps - cheers,
> > >> > > Tim
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > --
> > > Clayton Turner
> > > email: caturner3@g.cofc.edu <javascript:;>
> > > phone: (843)-424-3784
> > > web: claytonturner.blogspot.com
> > >
> > > ----------------------------------------------------------------------
> > > --------------------------- “When scientifically investigating the
> > > natural world, the only thing worse than a blind believer is a seeing
> > > denier.”
> > > - Neil deGrasse Tyson
> > >
> >
> >
> >
> > --
> > --
> > Clayton Turner
> > email: caturner3@g.cofc.edu <javascript:;>
> > phone: (843)-424-3784
> > web: claytonturner.blogspot.com
> >
> ----------------------------------------------------------------------------------------------
> > ---
> > “When scientifically investigating the natural world, the only thing
> worse than
> > a blind believer is a seeing denier.”
> > - Neil deGrasse Tyson
>
RE: Change from SNOMEDCT to SNOMEDCT_US affecting
v_snomed_fword_lookup
Posted by "Chen, Pei" <Pe...@childrens.harvard.edu>.
VJ,
Would it make sense to add in('SNOMEDCT_US') to the default ctakes-ytex/scripts/data/**/insert_view.sql?
That way it'll support umls2011 as well as the newer 2014 naming conventions?
Ex: inner join MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in ( 'SNOMEDCT', 'SNOMEDCT_US','RXNORM')
--Pei
> -----Original Message-----
> From: clayclay911@gmail.com [mailto:clayclay911@gmail.com] On Behalf Of
> Clayton Turner
> Sent: Thursday, August 21, 2014 4:25 PM
> To: dev@ctakes.apache.org
> Subject: Re: Change from SNOMEDCT to SNOMEDCT_US affecting
> v_snomed_fword_lookup
>
> Ah, I just switched to the ytex branch and all is good now. The SNOMED_US
> issue has been plaguing me for weeks now so thanks a million for that.
>
>
> On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner <ca...@g.cofc.edu>
> wrote:
>
> > Awesome. This is just what I needed for the longest time.
> >
> > I'm having a slight issue. When running either the ytex pipeline or
> > ytex version of the AggregatePlaintextUMLSProcessor I get an error
> > during initialization.
> >
> > My DictionaryLookupAnnotator.xml is raising a
> > org.apache.uima.resource.ResourceInitializationException causedby:
> > java.lang.ClassNotFoundException:
> > edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
> >
> > I feel like I may have drifted away from what I need, though, because
> > before this the CPE was complaining about a lack of
> > LookupDesc_SNOMED.xml file. I found a ytex version of this on a google
> > code site somewhere and pasted it where the CPE was looking for it. Now
> this error is coming up.
> >
> > Could my problem be solved with just a re-run of the ant script (was
> > just trying to avoid since it takes ages) or is it a different issue?
> >
> >
> > On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell
> > <ti...@gmail.com>
> > wrote:
> >
> >> Hi John,
> >>
> >> I'm not sure what was going on with the @db.schema@ error, although I
> >> was getting it as well before with my prior build of 3.1.2 - I assume
> >> that you've fixed something (thank you!) to make this go away. I
> >> rebuilt everything from scratch and it's working now.
> >>
> >> I think one other thing I had to change was that after I had finished
> >> the install/build, the cTakes version of LookupDesc_Db.xml doesn't
> >> work (in
> >> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I
> >> had to copy in an older version of the file from 3.1.1 to get the
> >> default cTakes AggregatePlaintextUMLSProcessor pipeline working,
> >> although please double-check that as my memory is a little foggy.
> >>
> >> But yes, here's what I have working since re-building:
> >> 1. ytex-pipeline.xml
> >> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
> >> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with
> >> swapping the LookupDesc_Db.xml file as above)
> >>
> >> I've even made modifications to the ytex version of
> >> LookupDesc_SNOMED.xml to get it tagging Disease Disorders, along with
> database modifications to
> >> have it store these entries as well, which is working great. Literally,
> >> everything is working perfectly now.
> >>
> >> Still so much for me to learn! Let me know if you need any more details.
> >>
> >> All the best,
> >> Tim
> >>
> >>
> >>
> >> On Tue, Aug 19, 2014 at 4:31 AM, John Green
> >> <jo...@gmail.com>
> >> wrote:
> >>
> >> > I have not had time to implement this - to clarify out of
> >> > curiosity,
> >> does
> >> > this clear up the @db.schema@ error Tim? And did you successfully
> >> > run ytex with the ctakes dictionary-lookup?
> >> >
> >> >
> >> > JG
> >> > —
> >> > Sent from Mailbox for iPhone
> >> >
> >> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell
> >> > <ti...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi folks,
> >> > > I was having an issue with the current build (from svn) of
> >> > > ctakes/ytex
> >> > not
> >> > > identifying any annotations as some folks on this board. I
> >> > > traced it
> >> to
> >> > > the fact that the UMLS database has at sometime in the relatively
> >> recent
> >> > > past changed the SAB tag in the MRCONSO table for SNOMED terms
> >> > > from SNOMEDCT to SNOMEDCT_US. I just had a newer version of
> UMLS
> >> > > that uses SNOMEDCT_US. Thus when the install script tried to
> >> > > create the v_snomed_fword_lookup table, it wasn't finding any of
> >> > > the SNOMEDCT
> >> terms,
> >> > > thus nothing was getting annotated.
> >> > > The ytex install script was just looking for things in MRCONSO
> >> > > with
> >> the
> >> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
> >> changing
> >> > > this to SNOMEDCT_US in the file
> >> > >
> >> >
> >> CTAKES_HOME/bin/ctakes-
> ytex/scripts/data/mysql/umls/insert_view_templ
> >> ate.sql
> >> > > it now works (for mysql users) to find the annotations. You can
> >> > > just
> >> > re-run
> >> > > the ytex setup script, but that takes hours - instead, I just
> >> > > deleted
> >> all
> >> > > the data from the v_snomed_fword_lookup table and basically ran
> >> > > the
> >> sql
> >> > > command to repopulate the table and it worked fine. Here's the
> >> > > code,
> >> n.b.
> >> > > my schema name for my umls database is 'umls' - change the code
> >> > > below
> >> if
> >> > > yours is different.
> >> > > delete from v_snomed_fword_lookup; insert into
> >> > > v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> >> > > stem_str)
> >> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> >> > > from umls_aui_fword c inner join umls.MRCONSO mrc on c.aui =
> >> > > mrc.aui and mrc.SAB in ( 'SNOMEDCT_US', 'RXNORM') inner join (
> >> > > select cui, min(tui) tui from umls.MRSTY sty where sty.tui in (
> >> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> >> > > 'T190', 'T191', 'T033',
> >> > > 'T184',
> >> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> >> > > 'T018', 'T021', 'T024',
> >> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> >> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> >> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> >> > > 'T110', 'T127',
> >> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> >> > > 'T074', 'T075',
> >> > > 'T059'
> >> > > )
> >> > > group by cui
> >> > > ) t on t.cui = mrc.cui
> >> > > ;
> >> > > Hope it helps - cheers,
> >> > > Tim
> >> >
> >>
> >
> >
> >
> > --
> > --
> > Clayton Turner
> > email: caturner3@g.cofc.edu
> > phone: (843)-424-3784
> > web: claytonturner.blogspot.com
> >
> > ----------------------------------------------------------------------
> > --------------------------- “When scientifically investigating the
> > natural world, the only thing worse than a blind believer is a seeing
> > denier.”
> > - Neil deGrasse Tyson
> >
>
>
>
> --
> --
> Clayton Turner
> email: caturner3@g.cofc.edu
> phone: (843)-424-3784
> web: claytonturner.blogspot.com
> ----------------------------------------------------------------------------------------------
> ---
> “When scientifically investigating the natural world, the only thing worse than
> a blind believer is a seeing denier.”
> - Neil deGrasse Tyson
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by Clayton Turner <ca...@g.cofc.edu>.
Ah, I just switched to the ytex branch and all is good now. The SNOMED_US
issue has been plaguing me for weeks now so thanks a million for that.
On Thu, Aug 21, 2014 at 2:13 PM, Clayton Turner <ca...@g.cofc.edu>
wrote:
> Awesome. This is just what I needed for the longest time.
>
> I'm having a slight issue. When running either the ytex pipeline or ytex
> version of the AggregatePlaintextUMLSProcessor I get an error during
> initialization.
>
> My DictionaryLookupAnnotator.xml is raising a
> org.apache.uima.resource.ResourceInitializationException causedby:
> java.lang.ClassNotFoundException:
> edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
>
> I feel like I may have drifted away from what I need, though, because
> before this the CPE was complaining about a lack of LookupDesc_SNOMED.xml
> file. I found a ytex version of this on a google code site somewhere and
> pasted it where the CPE was looking for it. Now this error is coming up.
>
> Could my problem be solved with just a re-run of the ant script (was just
> trying to avoid since it takes ages) or is it a different issue?
>
>
> On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell <ti...@gmail.com>
> wrote:
>
>> Hi John,
>>
>> I'm not sure what was going on with the @db.schema@ error, although I was
>> getting it as well before with my prior build of 3.1.2 - I assume that
>> you've fixed something (thank you!) to make this go away. I rebuilt
>> everything from scratch and it's working now.
>>
>> I think one other thing I had to change was that after I had finished the
>> install/build, the cTakes version of LookupDesc_Db.xml doesn't work (in
>> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I had to
>> copy in an older version of the file from 3.1.1 to get the default cTakes
>> AggregatePlaintextUMLSProcessor pipeline working, although please
>> double-check that as my memory is a little foggy.
>>
>> But yes, here's what I have working since re-building:
>> 1. ytex-pipeline.xml
>> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
>> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with swapping
>> the
>> LookupDesc_Db.xml file as above)
>>
>> I've even made modifications to the ytex version of LookupDesc_SNOMED.xml
>> to get it tagging Disease Disorders, along with database modifications to
>> have it store these entries as well, which is working great. Literally,
>> everything is working perfectly now.
>>
>> Still so much for me to learn! Let me know if you need any more details.
>>
>> All the best,
>> Tim
>>
>>
>>
>> On Tue, Aug 19, 2014 at 4:31 AM, John Green <jo...@gmail.com>
>> wrote:
>>
>> > I have not had time to implement this - to clarify out of curiosity,
>> does
>> > this clear up the @db.schema@ error Tim? And did you successfully run
>> > ytex with the ctakes dictionary-lookup?
>> >
>> >
>> > JG
>> > —
>> > Sent from Mailbox for iPhone
>> >
>> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
>> > wrote:
>> >
>> > > Hi folks,
>> > > I was having an issue with the current build (from svn) of ctakes/ytex
>> > not
>> > > identifying any annotations as some folks on this board. I traced it
>> to
>> > > the fact that the UMLS database has at sometime in the relatively
>> recent
>> > > past changed the SAB tag in the MRCONSO table for SNOMED terms from
>> > > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
>> > > SNOMEDCT_US. Thus when the install script tried to create the
>> > > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT
>> terms,
>> > > thus nothing was getting annotated.
>> > > The ytex install script was just looking for things in MRCONSO with
>> the
>> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
>> changing
>> > > this to SNOMEDCT_US in the file
>> > >
>> >
>> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
>> > > it now works (for mysql users) to find the annotations. You can just
>> > re-run
>> > > the ytex setup script, but that takes hours - instead, I just deleted
>> all
>> > > the data from the v_snomed_fword_lookup table and basically ran the
>> sql
>> > > command to repopulate the table and it worked fine. Here's the code,
>> n.b.
>> > > my schema name for my umls database is 'umls' - change the code below
>> if
>> > > yours is different.
>> > > delete from v_snomed_fword_lookup;
>> > > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
>> > > stem_str)
>> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
>> > > from umls_aui_fword c
>> > > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
>> > > 'SNOMEDCT_US', 'RXNORM')
>> > > inner join
>> > > (
>> > > select cui, min(tui) tui
>> > > from umls.MRSTY sty
>> > > where sty.tui in
>> > > (
>> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
>> > > 'T190', 'T191', 'T033',
>> > > 'T184',
>> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
>> > > 'T018', 'T021', 'T024',
>> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
>> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
>> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
>> > > 'T110', 'T127',
>> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
>> > > 'T074', 'T075',
>> > > 'T059'
>> > > )
>> > > group by cui
>> > > ) t on t.cui = mrc.cui
>> > > ;
>> > > Hope it helps - cheers,
>> > > Tim
>> >
>>
>
>
>
> --
> --
> Clayton Turner
> email: caturner3@g.cofc.edu
> phone: (843)-424-3784
> web: claytonturner.blogspot.com
>
> -------------------------------------------------------------------------------------------------
> “When scientifically investigating the natural world, the only thing worse
> than a blind believer is a seeing denier.”
> - Neil deGrasse Tyson
>
--
--
Clayton Turner
email: caturner3@g.cofc.edu
phone: (843)-424-3784
web: claytonturner.blogspot.com
-------------------------------------------------------------------------------------------------
“When scientifically investigating the natural world, the only thing worse
than a blind believer is a seeing denier.”
- Neil deGrasse Tyson
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by Clayton Turner <ca...@g.cofc.edu>.
Awesome. This is just what I needed for the longest time.
I'm having a slight issue. When running either the ytex pipeline or ytex
version of the AggregatePlaintextUMLSProcessor I get an error during
initialization.
My DictionaryLookupAnnotator.xml is raising a
org.apache.uima.resource.ResourceInitializationException causedby:
java.lang.ClassNotFoundException:
edu.mayo.bmi.uima.lookup.ae.FirstTokenPermLookupInitializerImpl
I feel like I may have drifted away from what I need, though, because
before this the CPE was complaining about a lack of LookupDesc_SNOMED.xml
file. I found a ytex version of this on a google code site somewhere and
pasted it where the CPE was looking for it. Now this error is coming up.
Could my problem be solved with just a re-run of the ant script (was just
trying to avoid since it takes ages) or is it a different issue?
On Tue, Aug 19, 2014 at 12:58 PM, Tim O'Connell <ti...@gmail.com>
wrote:
> Hi John,
>
> I'm not sure what was going on with the @db.schema@ error, although I was
> getting it as well before with my prior build of 3.1.2 - I assume that
> you've fixed something (thank you!) to make this go away. I rebuilt
> everything from scratch and it's working now.
>
> I think one other thing I had to change was that after I had finished the
> install/build, the cTakes version of LookupDesc_Db.xml doesn't work (in
> resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I had to
> copy in an older version of the file from 3.1.1 to get the default cTakes
> AggregatePlaintextUMLSProcessor pipeline working, although please
> double-check that as my memory is a little foggy.
>
> But yes, here's what I have working since re-building:
> 1. ytex-pipeline.xml
> 2. ytex version of AggregatePlaintextUMLSProcessor.xml
> 3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with swapping the
> LookupDesc_Db.xml file as above)
>
> I've even made modifications to the ytex version of LookupDesc_SNOMED.xml
> to get it tagging Disease Disorders, along with database modifications to
> have it store these entries as well, which is working great. Literally,
> everything is working perfectly now.
>
> Still so much for me to learn! Let me know if you need any more details.
>
> All the best,
> Tim
>
>
>
> On Tue, Aug 19, 2014 at 4:31 AM, John Green <jo...@gmail.com>
> wrote:
>
> > I have not had time to implement this - to clarify out of curiosity, does
> > this clear up the @db.schema@ error Tim? And did you successfully run
> > ytex with the ctakes dictionary-lookup?
> >
> >
> > JG
> > —
> > Sent from Mailbox for iPhone
> >
> > On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
> > wrote:
> >
> > > Hi folks,
> > > I was having an issue with the current build (from svn) of ctakes/ytex
> > not
> > > identifying any annotations as some folks on this board. I traced it
> to
> > > the fact that the UMLS database has at sometime in the relatively
> recent
> > > past changed the SAB tag in the MRCONSO table for SNOMED terms from
> > > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
> > > SNOMEDCT_US. Thus when the install script tried to create the
> > > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT
> terms,
> > > thus nothing was getting annotated.
> > > The ytex install script was just looking for things in MRCONSO with the
> > > SNOMEDCT SAB tag when it created the ytex lookup table - so, by
> changing
> > > this to SNOMEDCT_US in the file
> > >
> >
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> > > it now works (for mysql users) to find the annotations. You can just
> > re-run
> > > the ytex setup script, but that takes hours - instead, I just deleted
> all
> > > the data from the v_snomed_fword_lookup table and basically ran the sql
> > > command to repopulate the table and it worked fine. Here's the code,
> n.b.
> > > my schema name for my umls database is 'umls' - change the code below
> if
> > > yours is different.
> > > delete from v_snomed_fword_lookup;
> > > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> > > stem_str)
> > > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> > > from umls_aui_fword c
> > > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> > > 'SNOMEDCT_US', 'RXNORM')
> > > inner join
> > > (
> > > select cui, min(tui) tui
> > > from umls.MRSTY sty
> > > where sty.tui in
> > > (
> > > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> > > 'T190', 'T191', 'T033',
> > > 'T184',
> > > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> > > 'T018', 'T021', 'T024',
> > > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> > > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> > > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> > > 'T110', 'T127',
> > > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> > > 'T074', 'T075',
> > > 'T059'
> > > )
> > > group by cui
> > > ) t on t.cui = mrc.cui
> > > ;
> > > Hope it helps - cheers,
> > > Tim
> >
>
--
--
Clayton Turner
email: caturner3@g.cofc.edu
phone: (843)-424-3784
web: claytonturner.blogspot.com
-------------------------------------------------------------------------------------------------
“When scientifically investigating the natural world, the only thing worse
than a blind believer is a seeing denier.”
- Neil deGrasse Tyson
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by Tim O'Connell <ti...@gmail.com>.
Hi John,
I'm not sure what was going on with the @db.schema@ error, although I was
getting it as well before with my prior build of 3.1.2 - I assume that
you've fixed something (thank you!) to make this go away. I rebuilt
everything from scratch and it's working now.
I think one other thing I had to change was that after I had finished the
install/build, the cTakes version of LookupDesc_Db.xml doesn't work (in
resources\org\apache\ctakes\dictionary\lookup) - I'm pretty sure I had to
copy in an older version of the file from 3.1.1 to get the default cTakes
AggregatePlaintextUMLSProcessor pipeline working, although please
double-check that as my memory is a little foggy.
But yes, here's what I have working since re-building:
1. ytex-pipeline.xml
2. ytex version of AggregatePlaintextUMLSProcessor.xml
3. cTakes version of AggregatePlaintextUMLSProcessor.xml (with swapping the
LookupDesc_Db.xml file as above)
I've even made modifications to the ytex version of LookupDesc_SNOMED.xml
to get it tagging Disease Disorders, along with database modifications to
have it store these entries as well, which is working great. Literally,
everything is working perfectly now.
Still so much for me to learn! Let me know if you need any more details.
All the best,
Tim
On Tue, Aug 19, 2014 at 4:31 AM, John Green <jo...@gmail.com>
wrote:
> I have not had time to implement this - to clarify out of curiosity, does
> this clear up the @db.schema@ error Tim? And did you successfully run
> ytex with the ctakes dictionary-lookup?
>
>
> JG
> —
> Sent from Mailbox for iPhone
>
> On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
> wrote:
>
> > Hi folks,
> > I was having an issue with the current build (from svn) of ctakes/ytex
> not
> > identifying any annotations as some folks on this board. I traced it to
> > the fact that the UMLS database has at sometime in the relatively recent
> > past changed the SAB tag in the MRCONSO table for SNOMED terms from
> > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
> > SNOMEDCT_US. Thus when the install script tried to create the
> > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT terms,
> > thus nothing was getting annotated.
> > The ytex install script was just looking for things in MRCONSO with the
> > SNOMEDCT SAB tag when it created the ytex lookup table - so, by changing
> > this to SNOMEDCT_US in the file
> >
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> > it now works (for mysql users) to find the annotations. You can just
> re-run
> > the ytex setup script, but that takes hours - instead, I just deleted all
> > the data from the v_snomed_fword_lookup table and basically ran the sql
> > command to repopulate the table and it worked fine. Here's the code, n.b.
> > my schema name for my umls database is 'umls' - change the code below if
> > yours is different.
> > delete from v_snomed_fword_lookup;
> > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> > stem_str)
> > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> > from umls_aui_fword c
> > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> > 'SNOMEDCT_US', 'RXNORM')
> > inner join
> > (
> > select cui, min(tui) tui
> > from umls.MRSTY sty
> > where sty.tui in
> > (
> > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> > 'T190', 'T191', 'T033',
> > 'T184',
> > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> > 'T018', 'T021', 'T024',
> > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> > 'T110', 'T127',
> > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> > 'T074', 'T075',
> > 'T059'
> > )
> > group by cui
> > ) t on t.cui = mrc.cui
> > ;
> > Hope it helps - cheers,
> > Tim
>
Re: Change from SNOMEDCT to SNOMEDCT_US affecting
v_snomed_fword_lookup
Posted by John Green <jo...@gmail.com>.
I have not had time to implement this - to clarify out of curiosity, does this clear up the @db.schema@ error Tim? And did you successfully run ytex with the ctakes dictionary-lookup?
JG
—
Sent from Mailbox for iPhone
On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
wrote:
> Hi folks,
> I was having an issue with the current build (from svn) of ctakes/ytex not
> identifying any annotations as some folks on this board. I traced it to
> the fact that the UMLS database has at sometime in the relatively recent
> past changed the SAB tag in the MRCONSO table for SNOMED terms from
> SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
> SNOMEDCT_US. Thus when the install script tried to create the
> v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT terms,
> thus nothing was getting annotated.
> The ytex install script was just looking for things in MRCONSO with the
> SNOMEDCT SAB tag when it created the ytex lookup table - so, by changing
> this to SNOMEDCT_US in the file
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> it now works (for mysql users) to find the annotations. You can just re-run
> the ytex setup script, but that takes hours - instead, I just deleted all
> the data from the v_snomed_fword_lookup table and basically ran the sql
> command to repopulate the table and it worked fine. Here's the code, n.b.
> my schema name for my umls database is 'umls' - change the code below if
> yours is different.
> delete from v_snomed_fword_lookup;
> insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> stem_str)
> select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> from umls_aui_fword c
> inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> 'SNOMEDCT_US', 'RXNORM')
> inner join
> (
> select cui, min(tui) tui
> from umls.MRSTY sty
> where sty.tui in
> (
> 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> 'T190', 'T191', 'T033',
> 'T184',
> 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> 'T018', 'T021', 'T024',
> 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> 'T110', 'T127',
> 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> 'T074', 'T075',
> 'T059'
> )
> group by cui
> ) t on t.cui = mrc.cui
> ;
> Hope it helps - cheers,
> Tim
Re: Change from SNOMEDCT to SNOMEDCT_US affecting v_snomed_fword_lookup
Posted by Tim O'Connell <ti...@gmail.com>.
Thanks!
Tim
On Sat, Aug 16, 2014 at 9:07 AM, John Green <jo...@gmail.com>
wrote:
> Great catch!—
> Sent from Mailbox for iPhone
>
> On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
> wrote:
>
> > Hi folks,
> > I was having an issue with the current build (from svn) of ctakes/ytex
> not
> > identifying any annotations as some folks on this board. I traced it to
> > the fact that the UMLS database has at sometime in the relatively recent
> > past changed the SAB tag in the MRCONSO table for SNOMED terms from
> > SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
> > SNOMEDCT_US. Thus when the install script tried to create the
> > v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT terms,
> > thus nothing was getting annotated.
> > The ytex install script was just looking for things in MRCONSO with the
> > SNOMEDCT SAB tag when it created the ytex lookup table - so, by changing
> > this to SNOMEDCT_US in the file
> >
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> > it now works (for mysql users) to find the annotations. You can just
> re-run
> > the ytex setup script, but that takes hours - instead, I just deleted all
> > the data from the v_snomed_fword_lookup table and basically ran the sql
> > command to repopulate the table and it worked fine. Here's the code, n.b.
> > my schema name for my umls database is 'umls' - change the code below if
> > yours is different.
> > delete from v_snomed_fword_lookup;
> > insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> > stem_str)
> > select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> > from umls_aui_fword c
> > inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> > 'SNOMEDCT_US', 'RXNORM')
> > inner join
> > (
> > select cui, min(tui) tui
> > from umls.MRSTY sty
> > where sty.tui in
> > (
> > 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> > 'T190', 'T191', 'T033',
> > 'T184',
> > 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> > 'T018', 'T021', 'T024',
> > 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> > 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> > 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> > 'T110', 'T127',
> > 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> > 'T074', 'T075',
> > 'T059'
> > )
> > group by cui
> > ) t on t.cui = mrc.cui
> > ;
> > Hope it helps - cheers,
> > Tim
>
Re: Change from SNOMEDCT to SNOMEDCT_US affecting
v_snomed_fword_lookup
Posted by John Green <jo...@gmail.com>.
Great catch!—
Sent from Mailbox for iPhone
On Sat, Aug 16, 2014 at 2:53 AM, Tim O'Connell <ti...@gmail.com>
wrote:
> Hi folks,
> I was having an issue with the current build (from svn) of ctakes/ytex not
> identifying any annotations as some folks on this board. I traced it to
> the fact that the UMLS database has at sometime in the relatively recent
> past changed the SAB tag in the MRCONSO table for SNOMED terms from
> SNOMEDCT to SNOMEDCT_US. I just had a newer version of UMLS that uses
> SNOMEDCT_US. Thus when the install script tried to create the
> v_snomed_fword_lookup table, it wasn't finding any of the SNOMEDCT terms,
> thus nothing was getting annotated.
> The ytex install script was just looking for things in MRCONSO with the
> SNOMEDCT SAB tag when it created the ytex lookup table - so, by changing
> this to SNOMEDCT_US in the file
> CTAKES_HOME/bin/ctakes-ytex/scripts/data/mysql/umls/insert_view_template.sql
> it now works (for mysql users) to find the annotations. You can just re-run
> the ytex setup script, but that takes hours - instead, I just deleted all
> the data from the v_snomed_fword_lookup table and basically ran the sql
> command to repopulate the table and it worked fine. Here's the code, n.b.
> my schema name for my umls database is 'umls' - change the code below if
> yours is different.
> delete from v_snomed_fword_lookup;
> insert into v_snomed_fword_lookup (cui, tui, fword, fstem, tok_str,
> stem_str)
> select mrc.cui, t.tui, c.fword, c.fstem, c.tok_str, c.stem_str
> from umls_aui_fword c
> inner join umls.MRCONSO mrc on c.aui = mrc.aui and mrc.SAB in (
> 'SNOMEDCT_US', 'RXNORM')
> inner join
> (
> select cui, min(tui) tui
> from umls.MRSTY sty
> where sty.tui in
> (
> 'T019', 'T020', 'T037', 'T046', 'T047', 'T048', 'T049', 'T050',
> 'T190', 'T191', 'T033',
> 'T184',
> 'T017', 'T029', 'T023', 'T030', 'T031', 'T022', 'T025', 'T026',
> 'T018', 'T021', 'T024',
> 'T116', 'T195', 'T123', 'T122', 'T118', 'T103', 'T120', 'T104',
> 'T200', 'T111', 'T196', 'T126', 'T131', 'T125', 'T129', 'T130',
> 'T197', 'T119', 'T124', 'T114', 'T109', 'T115', 'T121', 'T192',
> 'T110', 'T127',
> 'T060', 'T065', 'T058', 'T059', 'T063', 'T062', 'T061',
> 'T074', 'T075',
> 'T059'
> )
> group by cui
> ) t on t.cui = mrc.cui
> ;
> Hope it helps - cheers,
> Tim