You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stanbol.apache.org by "Olivier Grisel (JIRA)" <ji...@apache.org> on 2011/09/02 17:35:09 UTC
[jira] [Created] (STANBOL-321) Named Entity detection engine should
better deal with hypenated text
Named Entity detection engine should better deal with hypenated text
--------------------------------------------------------------------
Key: STANBOL-321
URL: https://issues.apache.org/jira/browse/STANBOL-321
Project: Stanbol
Issue Type: Bug
Reporter: Olivier Grisel
Assignee: Olivier Grisel
We need some pre-processing to make it easier for OpenNLP to deal with hyphens, for instance this is an example of a real PDF:
Sparse RBMs and sparse auto-encoders
(RBM, SAE): In some of our experiments, we
train sparse RBMs (Hinton et al., 2006) and
sparse auto-encoders (Ranzato et al., 2007; Ben-
gio et al., 2006), both using a logistic sigmoid non-
linearity g(W x + b). These algorithms yield a set
of weights W and biases b. To obtain the dictio-
nary, D, we simply discard the biases and take
D = W ⊤ , then normalize the columns of D.
The current implementation return a TextAnnotation with entity-type = Person for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (STANBOL-321) Named Entity detection engine
should better deal with hyphenated text
Posted by "Olivier Grisel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125187#comment-13125187 ]
Olivier Grisel commented on STANBOL-321:
----------------------------------------
We should have a look at how solr.HyphenatedWordsFilterFactory works and whether we can reuse it directly for our purpose.
> Named Entity detection engine should better deal with hyphenated text
> ---------------------------------------------------------------------
>
> Key: STANBOL-321
> URL: https://issues.apache.org/jira/browse/STANBOL-321
> Project: Stanbol
> Issue Type: Bug
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
>
> We need some pre-processing to make it easier for OpenNLP to deal with hyphens, for instance this is an example of a real PDF:
> Sparse RBMs and sparse auto-encoders
> (RBM, SAE): In some of our experiments, we
> train sparse RBMs (Hinton et al., 2006) and
> sparse auto-encoders (Ranzato et al., 2007; Ben-
> gio et al., 2006), both using a logistic sigmoid non-
> linearity g(W x + b). These algorithms yield a set
> of weights W and biases b. To obtain the dictio-
> nary, D, we simply discard the biases and take
> D = W ⊤ , then normalize the columns of D.
> The current implementation return a TextAnnotation with entity-type = Person for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (STANBOL-321) Named Entity detection engine should
better deal with hyphenated text
Posted by "Olivier Grisel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivier Grisel updated STANBOL-321:
-----------------------------------
Summary: Named Entity detection engine should better deal with hyphenated text (was: Named Entity detection engine should better deal with hypenated text)
> Named Entity detection engine should better deal with hyphenated text
> ---------------------------------------------------------------------
>
> Key: STANBOL-321
> URL: https://issues.apache.org/jira/browse/STANBOL-321
> Project: Stanbol
> Issue Type: Bug
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
>
> We need some pre-processing to make it easier for OpenNLP to deal with hyphens, for instance this is an example of a real PDF:
> Sparse RBMs and sparse auto-encoders
> (RBM, SAE): In some of our experiments, we
> train sparse RBMs (Hinton et al., 2006) and
> sparse auto-encoders (Ranzato et al., 2007; Ben-
> gio et al., 2006), both using a logistic sigmoid non-
> linearity g(W x + b). These algorithms yield a set
> of weights W and biases b. To obtain the dictio-
> nary, D, we simply discard the biases and take
> D = W ⊤ , then normalize the columns of D.
> The current implementation return a TextAnnotation with entity-type = Person for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (STANBOL-321) Named Entity detection engine
should better deal with hyphenated text
Posted by "Joern Kottmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096705#comment-13096705 ]
Joern Kottmann commented on STANBOL-321:
----------------------------------------
We also have this issue when we analyze scanned news paper articles. In my opinion there should be a dedicated component to remove the hyphen and merge the tokens.
> Named Entity detection engine should better deal with hyphenated text
> ---------------------------------------------------------------------
>
> Key: STANBOL-321
> URL: https://issues.apache.org/jira/browse/STANBOL-321
> Project: Stanbol
> Issue Type: Bug
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
>
> We need some pre-processing to make it easier for OpenNLP to deal with hyphens, for instance this is an example of a real PDF:
> Sparse RBMs and sparse auto-encoders
> (RBM, SAE): In some of our experiments, we
> train sparse RBMs (Hinton et al., 2006) and
> sparse auto-encoders (Ranzato et al., 2007; Ben-
> gio et al., 2006), both using a logistic sigmoid non-
> linearity g(W x + b). These algorithms yield a set
> of weights W and biases b. To obtain the dictio-
> nary, D, we simply discard the biases and take
> D = W ⊤ , then normalize the columns of D.
> The current implementation return a TextAnnotation with entity-type = Person for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (STANBOL-321) Named Entity detection engine should
better deal with hyphenated text
Posted by "Fabian Christ (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabian Christ updated STANBOL-321:
----------------------------------
Issue Type: Improvement (was: Bug)
> Named Entity detection engine should better deal with hyphenated text
> ---------------------------------------------------------------------
>
> Key: STANBOL-321
> URL: https://issues.apache.org/jira/browse/STANBOL-321
> Project: Stanbol
> Issue Type: Improvement
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
>
> We need some pre-processing to make it easier for OpenNLP to deal with hyphens, for instance this is an example of a real PDF:
> Sparse RBMs and sparse auto-encoders
> (RBM, SAE): In some of our experiments, we
> train sparse RBMs (Hinton et al., 2006) and
> sparse auto-encoders (Ranzato et al., 2007; Ben-
> gio et al., 2006), both using a logistic sigmoid non-
> linearity g(W x + b). These algorithms yield a set
> of weights W and biases b. To obtain the dictio-
> nary, D, we simply discard the biases and take
> D = W ⊤ , then normalize the columns of D.
> The current implementation return a TextAnnotation with entity-type = Person for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira