You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ctakes.apache.org by "Johnsd11 (via GitHub)" <gi...@apache.org> on 2025/04/28 17:22:12 UTC
[I] [Question]: Initial CTakes analysis [ctakes]
Johnsd11 opened a new issue, #56:
URL: https://github.com/apache/ctakes/issues/56
### What’s your question?
I am looking for a NLP to read pathology reports and extract cancer related site, histology, stage and any other DX/RX data available. In looking at CTakes, I have a few questions;
- Is CTakes an appropriate tool to automate this task?
- The end goal would be a fully automated tool where text was presented to an API and data was returned.
- An added bonus, would be for the tool to annotate the text, so that a reviewer can more easily find the relevant data.
- For someone with a strong IT/software development background, but no NLP background what is the level of difficulty in getting started with this product?
### Context
_No response_
### What category does this question fall under?
None
### Contact Details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "pabramowitsch (via GitHub)" <gi...@apache.org>.
pabramowitsch commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2837378968
I've been using cTakes at UCSF for 7 years on millions of notes and am familiar with the task(s) you're suggesting. I'm guessing you're new to the system so my answers will be fairly high level.
- In general given an appropriate configuration, disease types, anatomical location, DX, RX data can be fairly accurately obtained from any report including Path reports using cTakes However staging, specific sites, lesion types and sizes etc are not well detected. We have experiments here at UCSF and also UCSD using some supplementary ML techniques and LLMS to do a better job of the latter data items.
- The API part has been implemented several times in the cTakes archive and I've done my own. See the included project ctakes-web-rest for one example. How you do it has a lot to do with the way you want to consume the output... as a raw CAS object or some new data structure to serve your needs. A raw CAS is very large so often it makes sense to create the output message as a new object you populate by scanning the CAS in the server process. The CAS is part of the Apache UIMA project which is integral to cTakes.
- We've written one, but it belongs to UCSF. There isn't one built into the open source code.
- As a system built mostly on deterministic methods such as pattern matching, regexp with some basic classification machinery etc it's not too difficult to get started. But without a clinical informatics background or NLP basics you'll have a serious ramp-up period to understand the components of the pipeline, the data types you're looking for and how to harvest them from the CAS, It's a pretty complex in-memory representation of the complete parse results. The Java infrastructure retains quite a bit of the 2010-2015 pluggable / dependency-injection design patterns, which, if you haven't encountered them, may also take time to understand. You'll also need to familiarize yourself with the dictionary structures, and customization.
- Best thing is to get started and allow yourself several months at least.
Peter
On Monday, April 28, 2025 at 10:22:32 AM PDT, Johnsd11 ***@***.***> wrote:
Johnsd11 created an issue (apache/ctakes#56)
What’s your question?
I am looking for a NLP to read pathology reports and extract cancer related site, histology, stage and any other DX/RX data available. In looking at CTakes, I have a few questions;
- Is CTakes an appropriate tool to automate this task?
- The end goal would be a fully automated tool where text was presented to an API and data was returned.
- An added bonus, would be for the tool to annotate the text, so that a reviewer can more easily find the relevant data.
- For someone with a strong IT/software development background, but no NLP background what is the level of difficulty in getting started with this product?
Context
No response
What category does this question fall under?
None
Contact Details
No response
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2835965082
> Out of the box, cTakes would get you part of the way there, but would require several types of customization to meet your requirements. All of these are the kind of customizations that most of us have had to do, so there's nothing new here, but they are not trivial. As I see it they fall into these categories.
>
> 1. getting familiar with the cTakes Application, pipeline, annotator and
> vocabulary ecosystem
>
> 2. choosing a vocabulary subset that gives the best coverage of the terms
> you are looking for
>
> 3. adding one or more custom dictionaries to add terms & synonyms that are
> not present -
>
> 4. maybe employing the anatomical site annotator in your pipeline
>
> 5. deciding how to harvest and structure the data you extract from the CAS
> object which all the annotators target
>
> 6. decide how to deploy the application (standalone?, webservices host?
> multi-instance? ). Many considerations go into this and greatly affect
> ability to scale. There is more than one architectural solution that will
> work and allow you to get to your "fully automated" goal, but you will need
> to implement that yourself.
>
>
> A hint about highlighting the text - all annotations carry text offsets so with these you can write code (usually JS and CSS) to do your highlighting. native cTakes does not have any graphical display functionality.
>
> Another hint learned from experience. If you have many large texts (say, 20kb and above with lots of potential terms to discover), you can achieve much better throughput by breaking these into smaller chunks at sentence boundaries and tweaking offsets accordingly as you reassemble the chunks. The memory requirements grow rapidly with the size of the note.
>
> In summary, a strong developer background is a good starting point. To that you'd want to add medical informatics, and experience with scalable architectures. cTakes is a great kernel to your system but be prepared to dive deep.
Thanks for the detailed and thoughtful explanation.
The easiest part for me to understand and work through would be #6. My MO for this sort of thing with both currently used in the existing target system are Windows services with associated DB queues and DLLs called from the application. The former for items which are not needed as part of the "real time" application and the latter for those which are.
I currently have a homegrown application which looks for keywords and negation modifiers within a certain distance from the keywords which works moderately well.
My ignorance regarding NLP algorithms like CTakes is whether it is keyword driven, or it is self learning. If it is the latter, I have a fairly large collection of human curated data which I could feed a training module.
Where can I find an "executive overview" (30,000 foot view) of how the CTakes works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 closed issue #56: [Question]: Initial CTakes analysis
URL: https://github.com/apache/ctakes/issues/56
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2835961166
> I am looking for a NLP to read pathology reports and extract cancer
> related site, histology, stage and any other DX/RX data available. In
> looking at CTakes, I have a few questions;
>
> - Is CTakes an appropriate tool to automate this task?
I wrote a commercial surgical-pathology coding module some years ago, and
could imagine doing it in cTAKES.
Here's my two cents to add to the wealth of information Peter has already
provided.
Best luck.
> Where can I find an "executive overview" (30,000 foot view) of how the
CTakes works?
As Peter said, there's a lot of documentation out there!
Videos here: https://ctakes.apache.org/tutorials.html
Key point: it's built on top of UIMA https://uima.apache.org//
which ingests and annotates data from any source, letting you mix, match
and create your own annotators to build chains of analyses.
The cTAKES value-adds include a clinical type system and a spiffy
dictionary (see below).
> My ignorance regarding NLP algorithms like CTakes is whether it is
keyword driven, or it is self learning.
cTAKES is *not* "self-learning"; you have to tell it exactly what
information you want to extract from where.
Pro: High precision; explainable; you won't get the right answer for the
wrong reason.
Con: Low recall; brittle; you may not get answers at all! If you're
processing unpredictable document formats from many different facilities,
it can be hard to generalize over them.
> I currently have a homegrown application which looks for keywords and
negation modifiers within a certain distance from the keywords
cTAKES can certainly help with that.
-
*Keywords *cTAKES lets you use the NLM's UMLS Metathesaurus
<https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/index.html>,
using the dictionary framework Peter mentioned:
https://cwiki.apache.org/confluence/display/CTAKES/cTAKES+4.0+-+Fast+Dictionary+Lookup
These sources may be useful in building your custom dictionary:
- the NCI Thesaurus:
https://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/NCI/index.html
- CPT, if you want codes from there:
https://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/CPT/index.html
- For anatomy, I'm not familiar with the "anatomical site annotator"
Peter alludes to, but the FMA is better structured than SNOMED:
https://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/FMA/index.html
- *Negation*
Several annotators available:
https://cwiki.apache.org/confluence/display/CTAKES/cTAKES+4.0+-+Negation+Annotators
Distance-from-keywords is a start, but sentence detection and shallow
parsing both help.
I like the ctakes-ytex-uima NegexAnnotator and SentenceDetector.
-
*Document structure *I found header detection to be crucial in processing
pathology reports:
tracking specimens through a document, extracting tumor info from
tables, etc.
The cTAKES RegexSectionizer might work for you.
https://ctakes.apache.org/apidocs/4.0.0/org/apache/ctakes/core/ae/RegexSectionizer.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2835965980
1. The cTakes ecosystem is Java with a some optional Python code. I have
little experience running it in a Windows environment and so perhaps
someone else in the group can give you pointers. My instinct would be to
run it in a Linux based Docker instance - which I do anyway for some
clients. You can package it yourself as a standalone application talking
to a database or you can use a Webservice wrapper around it which exists in
the codebase (that is either Dockerized or packaged as.a WAR or both).
Then you can implement a REST client in a pure Windows environment if that
is easier for you.
2. cTakes is an open source project going back to 2012, and as such, uses
many different technical approaches in its various components: pattern
recognition, state machines, POS and treebank extractors and some ML
techniques but it does not have a user friendly training mechanism for
those components, although there are some examples.
The best way to understand it is to download it and get started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2835960043
Out of the box, cTakes would get you part of the way there, but would
require several types of customization to meet your requirements. All of
these are the kind of customizations that most of us have had to do, so
there's nothing new here, but they are not trivial. As I see it they fall
into these categories.
1. getting familiar with the cTakes Application, pipeline, annotator and
vocabulary ecosystem
2. choosing a vocabulary subset that gives the best coverage of the terms
you are looking for
3. adding one or more custom dictionaries to add terms & synonyms that are
not present -
4. maybe employing the anatomical site annotator in your pipeline
5. deciding how to harvest and structure the data you extract from the CAS
object which all the annotators target
6. decide how to deploy the application (standalone?, webservices host?
multi-instance? ). Many considerations go into this and greatly affect
ability to scale. There is more than one architectural solution that will
work and allow you to get to your "fully automated" goal, but you will need
to implement that yourself.
A hint about highlighting the text - all annotations carry text offsets so
with these you can write code (usually JS and CSS) to do your
highlighting. native cTakes does not have any graphical display
functionality.
Another hint learned from experience. If you have many large texts (say,
20kb and above with lots of potential terms to discover), you can achieve
much better throughput by breaking these into smaller chunks at sentence
boundaries and tweaking offsets accordingly as you reassemble the chunks.
The memory requirements grow rapidly with the size of the note.
In summary, a strong developer background is a good starting point. To
that you'd want to add medical informatics, and experience with scalable
architectures. cTakes is a great kernel to your system but be prepared to
dive deep.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question]: Initial CTakes analysis [ctakes]
Posted by "Johnsd11 (via GitHub)" <gi...@apache.org>.
Johnsd11 commented on issue #56:
URL: https://github.com/apache/ctakes/issues/56#issuecomment-2835966408
There are many images and papers online that describe cTakes at a high
level. Probably none of them are 100% comprehensive but they will get you
started.
try this:
https://cwiki.apache.org/confluence/display/CTAKES/cTAKES+4.0+-+Fast+Dictionary+Lookup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@ctakes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org