You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ctakes.apache.org by William Karl Thompson <wk...@northwestern.edu> on 2013/08/30 00:07:09 UTC
resources for training modules
Hello all,
I'm interested in training the relation extractor on some annotated notes from Northwestern clinical data, and I understand that cleartk is currently being used for this purpose in the cTAKES project. Could someone provide some pointers on how to go about using cleartk to train models that can then be invoked by a cTAKES module? Again, my focus for now is on the relation extractor. In case it's relevant, I'm intending to use the brat rapid annotation tool (http://brat.nlplab.org/) to generate a gold standard corpus.
Cheers,
Will
RE: resources for training modules
Posted by William Karl Thompson <wk...@northwestern.edu>.
Steve and Dima,
Thanks very much for these replies, this is very helpful to get started. I'll give this a try over the next couple of weeks, will let you know how it works.
Cheers,
Will
-----Original Message-----
From: Steven Bethard [mailto:stevenbethard@apache.org]
Sent: Friday, August 30, 2013 9:37 AM
To: dev@ctakes.apache.org
Subject: Re: resources for training modules
On Fri, Aug 30, 2013 at 8:46 AM, Dmitriy Dligach <dm...@childrens.harvard.edu> wrote:
> Retraining the relation extractor should be fairly easy. The
> instructions I am about to give you apply if you are using cTAKES 3.0.
> However, if you are planning to use the trunk version, my instructions
> may no longer be accurate. Relation extraction has undergone some
> changes recently in connection with cTAKES-190 issue and I don't fully
> understand these most recent changes yet (but I am working on it).
With the trunk version, there's no need to run PreprocessAndWriteXmi.
Just run RelationExtractorEvaluation or RelationExtractorTrain directly. (The XMIs will be automatically written to target/xmi.) I believe the only required argument is --batches-dir, which gives the directory containing the directories containing Knowtator_XML directories. The other (optional) arguments should be similar to what Dima described (and you can see what they are by looking at the static Options classes (and their superclasses) in RelationExtractorEvaluation and RelationExtractorTrain).
> If you are planning to annotate your data, it might be easier to use
> Knowtator since we already have a gold standard reader for Knowtator.
> If you want to use a different annotation tool, you just have to make
> sure you add the manual annotations to the gold view of the XMI files.
In the trunk version, most of the SHARP-specific stuff is handled by the SHARPXMI class. So if you need to customize things away from what was done for SHARP, that's probably where you'll need to go. At the moment, RelationExtractorTrain and RelationExtractorEvaluation call static methods on SHARPXMI, which means that it's not very extensible.
We could conceivably change these methods to non-static methods, and then extensions of relation-extractor could provide their own implementation. We're certainly open to modifying the infrastructure here, so if you have any suggestions please do pass them on.
Steve
Re: resources for training modules
Posted by Steven Bethard <st...@apache.org>.
On Fri, Aug 30, 2013 at 8:46 AM, Dmitriy Dligach
<dm...@childrens.harvard.edu> wrote:
> Retraining the relation extractor should be fairly easy. The instructions I
> am about to give you apply if you are using cTAKES 3.0. However, if you are
> planning to use the trunk version, my instructions may no longer be
> accurate. Relation extraction has undergone some changes recently in
> connection with cTAKES-190 issue and I don't fully understand these most
> recent changes yet (but I am working on it).
With the trunk version, there's no need to run PreprocessAndWriteXmi.
Just run RelationExtractorEvaluation or RelationExtractorTrain
directly. (The XMIs will be automatically written to target/xmi.) I
believe the only required argument is --batches-dir, which gives the
directory containing the directories containing Knowtator_XML
directories. The other (optional) arguments should be similar to what
Dima described (and you can see what they are by looking at the static
Options classes (and their superclasses) in
RelationExtractorEvaluation and RelationExtractorTrain).
> If you are planning to annotate your data, it might be easier to use
> Knowtator since we already have a gold standard reader for Knowtator. If you
> want to use a different annotation tool, you just have to make sure you add
> the manual annotations to the gold view of the XMI files.
In the trunk version, most of the SHARP-specific stuff is handled by
the SHARPXMI class. So if you need to customize things away from what
was done for SHARP, that's probably where you'll need to go. At the
moment, RelationExtractorTrain and RelationExtractorEvaluation call
static methods on SHARPXMI, which means that it's not very extensible.
We could conceivably change these methods to non-static methods, and
then extensions of relation-extractor could provide their own
implementation. We're certainly open to modifying the infrastructure
here, so if you have any suggestions please do pass them on.
Steve
Re: resources for training modules
Posted by Dmitriy Dligach <dm...@childrens.harvard.edu>.
Hi Will,
Retraining the relation extractor should be fairly easy. The
instructions I am about to give you apply if you are using cTAKES 3.0.
However, if you are planning to use the trunk version, my instructions
may no longer be accurate. Relation extraction has undergone some
changes recently in connection with cTAKES-190 issue and I don't fully
understand these most recent changes yet (but I am working on it).
1. Run PreprocessAndWriteXmi in the eval package, specifying the
location of the text of the notes, the location of the gold standard
relation
annotations, and the output directory. This class will run all the
preprocessing that is required for relation extraction and add gold standard
relation annotations to the CAS. The resulting CASes will be saved to
disk as XMI files.
2. Run RelationExtractorEvaluation, passing it the location of the XMI
files obtained in the previous steps and --grid-search option. This
class will use the annotations in the XMI files to find the optimal
training parameters using grid search and n-fold cross-validation. After
the execution completes, record the best set of parameters found by the
grid search. If you don't have a lot of time, this step can be skipped
(you can just use the default SVM parameters).
3. Update the model parameters in the main() method of
RelationExtractorTrain (pipelines package) to the values found by the
grid search. Run RelationExtractorTrain, specifying the location of the
XMI files. This class will (a) create a model that is necessary for
deployment of the relation module, and (b) create the descriptor files
which will ensure that the the relation AEs can be used as a part of a
UIMA pipeline.
If you are planning to annotate your data, it might be easier to use
Knowtator since we already have a gold standard reader for Knowtator. If
you want to use a different annotation tool, you just have to make sure
you add the manual annotations to the gold view of the XMI files. The
relation extractor reads the gold standard annotations from the gold view.
Hope this helps,
Dima
On 08/29/2013 06:07 PM, William Karl Thompson wrote:
> Hello all,
>
> I'm interested in training the relation extractor on some annotated notes from Northwestern clinical data, and I understand that cleartk is currently being used for this purpose in the cTAKES project. Could someone provide some pointers on how to go about using cleartk to train models that can then be invoked by a cTAKES module? Again, my focus for now is on the relation extractor. In case it's relevant, I'm intending to use the brat rapid annotation tool (http://brat.nlplab.org/) to generate a gold standard corpus.
>
> Cheers,
>
> Will
>
--
Dmitriy Dligach, PhD
Research Fellow
Children's Hospital Informatics Program
Boston Children's Hospital and Harvard Medical School
(617) 919-3596
RE: resources for training modules
Posted by William Karl Thompson <wk...@northwestern.edu>.
Thanks for the tip! I'll check it out (literally).
-----Original Message-----
From: Jörn Kottmann [mailto:kottmann@gmail.com]
Sent: Monday, September 02, 2013 3:14 AM
To: dev@ctakes.apache.org
Subject: Re: resources for training modules
On 08/30/2013 12:07 AM, William Karl Thompson wrote:
> In case it's relevant, I'm intending to use the brat rapid annotation tool (http://brat.nlplab.org/) to generate a gold standard corpus.
The OpenNLP trunk contains parsing code for brat. It should be fairly easy to use it to produce XMI files for the cTAKES trainer AE.
Jörn
Re: resources for training modules
Posted by Jörn Kottmann <ko...@gmail.com>.
On 08/30/2013 12:07 AM, William Karl Thompson wrote:
> In case it's relevant, I'm intending to use the brat rapid annotation tool (http://brat.nlplab.org/) to generate a gold standard corpus.
The OpenNLP trunk contains parsing code for brat. It should be fairly
easy to use it to produce XMI files for the cTAKES trainer AE.
Jörn