You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Яков Керанчук <zl...@gmail.com> on 2013/05/23 14:04:08 UTC
NER Money example
Hi all!
I'm experimenting with NER model en-ner-money.bin and trying to recognize
some strings like this: "twenty thousand dollars ", "10000 usd" and etc.
So result Span set is empty, trough en-ner-person is woking well on Person
examples.
Is there any correction for work with money, examples?
--
Best regards,
Yakov
Re: NER Money example
Posted by Яков Керанчук <zl...@gmail.com>.
Thanks for the advice, will do own model
On Thu, May 23, 2013 at 5:25 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> No there are no differences in your samples. Try to use capital USD
> instead of usd.
>
> The model was trained on English news text from the 90s try to give it
> some (old) news
> articles for testing.
>
> Jörn
>
>
> On 05/23/2013 03:16 PM, Яков Керанчук wrote:
>
>> Experimenting shows next results: only $ marked digits determined as money
>>
>> ------sentences: [The drop last week unwound most of the prior week's
>> jump,
>> suggesting employers were not laying off workers in response to tighter
>> fiscal policy, especially the $85 billion in across-the-board government
>> spending cuts that have dampened factory activity]
>> ------tokenizing
>> ------finding money
>> [[29..32) money]
>> [29..32) money
>> prepare model
>> ------sentences: [buy milk $2]
>> ------tokenizing
>> buy
>> milk
>> $
>> 2
>> ------finding money
>> [[2..4) money]
>> [2..4) money
>> ------pos tagging
>> VB
>> NN
>> $
>> CD
>> ------saving message to database
>> prepare model
>> ------sentences: [buy milk usd 2]
>> ------tokenizing
>> buy
>> milk
>> usd
>> 2
>> ------finding money
>> []
>> ------pos tagging
>> VB
>> NN
>> CD
>> ------saving message to database
>> prepare model
>> ------sentences: [Buy milk two Dollars]
>> ------tokenizing
>> Buy
>> milk
>> two
>> Dollars
>> ------finding money
>> []
>>
>> I have not noticed difference between SimpleTokenizer and TokenizerME in
>> this case
>>
>>
>> On Thu, May 23, 2013 at 5:00 PM, Jörn Kottmann <ko...@gmail.com>
>> wrote:
>>
>> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>>>
>>> Thanks for suggestion with own model, I'll try
>>>>
>>>> I use standard en-token.bin model, text contains mixed upper-lower case
>>>> words.
>>>>
>>>> For the english model you should use the SimpleTokenizer, the token
>>> output
>>> from the en-token.bin model is not compatible with the training data.
>>>
>>> Jörn
>>>
>>>
>>
>>
>
--
Best regards,
Yakov Keranchuk
+79263768032
Re: NER Money example
Posted by Jörn Kottmann <ko...@gmail.com>.
No there are no differences in your samples. Try to use capital USD
instead of usd.
The model was trained on English news text from the 90s try to give it
some (old) news
articles for testing.
Jörn
On 05/23/2013 03:16 PM, Яков Керанчук wrote:
> Experimenting shows next results: only $ marked digits determined as money
>
> ------sentences: [The drop last week unwound most of the prior week's jump,
> suggesting employers were not laying off workers in response to tighter
> fiscal policy, especially the $85 billion in across-the-board government
> spending cuts that have dampened factory activity]
> ------tokenizing
> ------finding money
> [[29..32) money]
> [29..32) money
> prepare model
> ------sentences: [buy milk $2]
> ------tokenizing
> buy
> milk
> $
> 2
> ------finding money
> [[2..4) money]
> [2..4) money
> ------pos tagging
> VB
> NN
> $
> CD
> ------saving message to database
> prepare model
> ------sentences: [buy milk usd 2]
> ------tokenizing
> buy
> milk
> usd
> 2
> ------finding money
> []
> ------pos tagging
> VB
> NN
> CD
> ------saving message to database
> prepare model
> ------sentences: [Buy milk two Dollars]
> ------tokenizing
> Buy
> milk
> two
> Dollars
> ------finding money
> []
>
> I have not noticed difference between SimpleTokenizer and TokenizerME in
> this case
>
>
> On Thu, May 23, 2013 at 5:00 PM, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>>
>>> Thanks for suggestion with own model, I'll try
>>>
>>> I use standard en-token.bin model, text contains mixed upper-lower case
>>> words.
>>>
>> For the english model you should use the SimpleTokenizer, the token output
>> from the en-token.bin model is not compatible with the training data.
>>
>> Jörn
>>
>
>
Re: NER Money example
Posted by Яков Керанчук <zl...@gmail.com>.
Experimenting shows next results: only $ marked digits determined as money
------sentences: [The drop last week unwound most of the prior week's jump,
suggesting employers were not laying off workers in response to tighter
fiscal policy, especially the $85 billion in across-the-board government
spending cuts that have dampened factory activity]
------tokenizing
------finding money
[[29..32) money]
[29..32) money
prepare model
------sentences: [buy milk $2]
------tokenizing
buy
milk
$
2
------finding money
[[2..4) money]
[2..4) money
------pos tagging
VB
NN
$
CD
------saving message to database
prepare model
------sentences: [buy milk usd 2]
------tokenizing
buy
milk
usd
2
------finding money
[]
------pos tagging
VB
NN
CD
------saving message to database
prepare model
------sentences: [Buy milk two Dollars]
------tokenizing
Buy
milk
two
Dollars
------finding money
[]
I have not noticed difference between SimpleTokenizer and TokenizerME in
this case
On Thu, May 23, 2013 at 5:00 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>
>> Thanks for suggestion with own model, I'll try
>>
>> I use standard en-token.bin model, text contains mixed upper-lower case
>> words.
>>
>
> For the english model you should use the SimpleTokenizer, the token output
> from the en-token.bin model is not compatible with the training data.
>
> Jörn
>
--
Best regards,
Yakov Keranchuk
+79263768032
Re: NER Money example
Posted by Lance Norskog <go...@gmail.com>.
Please add this to the documentation.
On 05/23/2013 06:22 AM, Jörn Kottmann wrote:
> Only for the English NER models on the SourceForge page.
>
> Jörn
>
> On 05/23/2013 03:13 PM, Ryan Josal wrote:
>> Is this true for the rest of the NER models too?
>>
>> Ryan
>>
>> On May 23, 2013, at 6:00, Jörn Kottmann <ko...@gmail.com> wrote:
>>
>>> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>>>> Thanks for suggestion with own model, I'll try
>>>>
>>>> I use standard en-token.bin model, text contains mixed upper-lower
>>>> case
>>>> words.
>>> For the english model you should use the SimpleTokenizer, the token
>>> output
>>> from the en-token.bin model is not compatible with the training data.
>>>
>>> Jörn
>
Re: NER Money example
Posted by Jörn Kottmann <ko...@gmail.com>.
Only for the English NER models on the SourceForge page.
Jörn
On 05/23/2013 03:13 PM, Ryan Josal wrote:
> Is this true for the rest of the NER models too?
>
> Ryan
>
> On May 23, 2013, at 6:00, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>>> Thanks for suggestion with own model, I'll try
>>>
>>> I use standard en-token.bin model, text contains mixed upper-lower case
>>> words.
>> For the english model you should use the SimpleTokenizer, the token output
>> from the en-token.bin model is not compatible with the training data.
>>
>> Jörn
Re: NER Money example
Posted by Ryan Josal <rj...@gmail.com>.
Is this true for the rest of the NER models too?
Ryan
On May 23, 2013, at 6:00, Jörn Kottmann <ko...@gmail.com> wrote:
> On 05/23/2013 02:56 PM, Яков Керанчук wrote:
>> Thanks for suggestion with own model, I'll try
>>
>> I use standard en-token.bin model, text contains mixed upper-lower case
>> words.
>
> For the english model you should use the SimpleTokenizer, the token output
> from the en-token.bin model is not compatible with the training data.
>
> Jörn
Re: NER Money example
Posted by Jörn Kottmann <ko...@gmail.com>.
On 05/23/2013 02:56 PM, Яков Керанчук wrote:
> Thanks for suggestion with own model, I'll try
>
> I use standard en-token.bin model, text contains mixed upper-lower case
> words.
For the english model you should use the SimpleTokenizer, the token output
from the en-token.bin model is not compatible with the training data.
Jörn
Re: NER Money example
Posted by Яков Керанчук <zl...@gmail.com>.
Thanks for suggestion with own model, I'll try
I use standard en-token.bin model, text contains mixed upper-lower case
words.
On Thu, May 23, 2013 at 4:23 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 05/23/2013 02:04 PM, Яков Керанчук wrote:
>
>> I'm experimenting with NER model en-ner-money.bin and trying to recognize
>> some strings like this: "twenty thousand dollars ", "10000 usd" and etc.
>>
>> So result Span set is empty, trough en-ner-person is woking well on Person
>> examples.
>>
>> Is there any correction for work with money, examples?
>>
>
> You should probably annotate some of your texts and then train on them,
> the money detection works well in my private data with my own model.
>
> Anyway, it should detect some money values, maybe a problem with the
> tokenization?
> How do you tokenize? Is all you text lower case? If yes that will not work
> well with the supplied
> model.
>
> Jörn
>
--
Best regards,
Yakov Keranchuk
+79263768032
Re: NER Money example
Posted by Jörn Kottmann <ko...@gmail.com>.
On 05/23/2013 02:04 PM, Яков Керанчук wrote:
> I'm experimenting with NER model en-ner-money.bin and trying to recognize
> some strings like this: "twenty thousand dollars ", "10000 usd" and etc.
>
> So result Span set is empty, trough en-ner-person is woking well on Person
> examples.
>
> Is there any correction for work with money, examples?
You should probably annotate some of your texts and then train on them,
the money detection works well in my private data with my own model.
Anyway, it should detect some money values, maybe a problem with the
tokenization?
How do you tokenize? Is all you text lower case? If yes that will not
work well with the supplied
model.
Jörn