You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by Jason Baldridge <ja...@gmail.com> on 2011/04/14 05:15:19 UTC
universal tagset
For many applications, it would be useful to have a universal tagset for any
language you are working with. See below for details on a project that
provides mappings from many standard treebanks to a course-grained tagset
(12 tags). We might want to support these mappings to simple tags in our
models (e.g. have a model that uses corpus-native tags and another that uses
universal tags).
Jason
--
Jason Baldridge
Assistant Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge
Hi everyone,
some of you have already heard about our universal part-of-speech
tagset (and are even using it), to others this might be new.
We sat down and read through the annotation guidelines of 25 treebanks
and created a mapping to an universal set of 12 coarse-grained
part-of-speech categories. We have described the tagset and
illustrated some use cases in a short write-up (see attached pdf).
Additionally, we have uploaded the mappings to a code repository with
version control so that new languages can be added or modification can
be made if necessary:
http://code.google.com/p/universal-pos-tags/
The paper is for now on arXiv:
http://arxiv.org/abs/1104.2086
We hope that you will find this resource useful for your own work. Let
us know if you have any comments,
Cheers,
Dipanjan, Ryan & Slav
Re: universal tagset
Posted by Jason Baldridge <ja...@gmail.com>.
Yep, that's right. -j
On Fri, Apr 15, 2011 at 9:05 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 4/14/11 5:15 AM, Jason Baldridge wrote:
>
>> For many applications, it would be useful to have a universal tagset for
>> any
>> language you are working with. See below for details on a project that
>> provides mappings from many standard treebanks to a course-grained tagset
>> (12 tags). We might want to support these mappings to simple tags in our
>> models (e.g. have a model that uses corpus-native tags and another that
>> uses
>> universal tags).
>>
>
> All we need to do is, to replace the pos-tags in the training data with the
> universal one, right? I guess we could add support for this to the
> converter tool or make a small tool which can replace the tags.
>
> Jörn
>
--
Jason Baldridge
Assistant Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge
Re: universal tagset
Posted by Jörn Kottmann <ko...@gmail.com>.
On 4/14/11 5:15 AM, Jason Baldridge wrote:
> For many applications, it would be useful to have a universal tagset for any
> language you are working with. See below for details on a project that
> provides mappings from many standard treebanks to a course-grained tagset
> (12 tags). We might want to support these mappings to simple tags in our
> models (e.g. have a model that uses corpus-native tags and another that uses
> universal tags).
All we need to do is, to replace the pos-tags in the training data with the
universal one, right? I guess we could add support for this to the
converter tool or make a small tool which can replace the tags.
Jörn
Re: universal tagset
Posted by Jason Baldridge <ja...@gmail.com>.
I'm not sure what you mean? The NER tags are a standard set of B-PER, I-PER,
etc for all languages, so it is small and universal.
Note that if one did want to do NER with the universal POS tagset (since POS
tagging is done before NER), you'd need to map the POS tags, train the
universal POS tagger, and then learn the NER.
On Thu, Apr 14, 2011 at 10:56 PM, James Kosin <ja...@gmail.com> wrote:
> On 4/13/2011 11:15 PM, Jason Baldridge wrote:
> > For many applications, it would be useful to have a universal tagset for
> any
> > language you are working with. See below for details on a project that
> > provides mappings from many standard treebanks to a course-grained tagset
> > (12 tags). We might want to support these mappings to simple tags in our
> > models (e.g. have a model that uses corpus-native tags and another that
> uses
> > universal tags).
> >
> > Jason
> >
> Jason,
>
> That looks like a good idea. Do they have tags for the name-finder as
> well or did they just do POS tagging?
>
> James
>
--
Jason Baldridge
Assistant Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge
Re: universal tagset
Posted by James Kosin <ja...@gmail.com>.
On 4/13/2011 11:15 PM, Jason Baldridge wrote:
> For many applications, it would be useful to have a universal tagset for any
> language you are working with. See below for details on a project that
> provides mappings from many standard treebanks to a course-grained tagset
> (12 tags). We might want to support these mappings to simple tags in our
> models (e.g. have a model that uses corpus-native tags and another that uses
> universal tags).
>
> Jason
>
Jason,
That looks like a good idea. Do they have tags for the name-finder as
well or did they just do POS tagging?
James