You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stanbol.apache.org by "Pablo Mendes (JIRA)" <ji...@apache.org> on 2012/06/04 12:49:25 UTC
[jira] [Created] (STANBOL-652) Benchmark should report evaluation
summary
Pablo Mendes created STANBOL-652:
------------------------------------
Summary: Benchmark should report evaluation summary
Key: STANBOL-652
URL: https://issues.apache.org/jira/browse/STANBOL-652
Project: Stanbol
Issue Type: Improvement
Components: Testing
Reporter: Pablo Mendes
Priority: Minor
The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (STANBOL-652) Benchmark should report evaluation
summary
Posted by "Pablo Mendes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401422#comment-13401422 ]
Pablo Mendes commented on STANBOL-652:
--------------------------------------
I like it a lot.
Saving the evaluation results to an RDF store would then allow one to slice-and-dice the results however he/she wants.
For example, how many correct results for person annotations?
SELECT count(?result)
WHERE {
?result sbc:state sbc:benchmark-state-succeeded .
?result sbc:about ?annotation .
?annotation entity ?person .
?person a dbpedia:Person .
}
Sharing this RDF via a SPARQL endpoint would then allow prompt Web-based report generation (aka visualization of results) with something like Sgvizler, for example
http://code.google.com/p/sgvizler/
Although what I'd actually do would be to get a CSV from this and use R to analyze the results. But even getting this CSV should be trivial from the RDF.
> Benchmark should report evaluation summary
> ------------------------------------------
>
> Key: STANBOL-652
> URL: https://issues.apache.org/jira/browse/STANBOL-652
> Project: Stanbol
> Issue Type: Improvement
> Components: Testing
> Reporter: Pablo Mendes
> Priority: Minor
> Labels: benchmark, evaluation
>
> The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
> dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (STANBOL-652) Benchmark should report evaluation
summary
Posted by "Rupert Westenthaler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/STANBOL-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401389#comment-13401389 ]
Rupert Westenthaler commented on STANBOL-652:
---------------------------------------------
Hi Pablo
I think the best think to start would be to refactor the benchmarking module so that is can return Benchmark results as RDF
I would even suggest to convert the Benchmarks to RDF (but still supporting the Text based format currently used)
urn:benchmark1 a sbc:Benchmark
sbc:content "The INPUT text"
sbc:expects urn:annotation1
sbc:expects urn:annotation2
sbc:complain urn:annotation3
urn:annotation1 a sbc:Annotation
entity <entity-URI>
NOTE: we might add different sbc:Annotation types in future (e.g. to validate Named Entities detection by NER tools or the detection of the correct language)
Results of a Benchmark execution
urn:benchmarkExecution1 a BenchmarkExecution
sbc:benchmark urn:benchmark1
sbc:num-expects 2^^xsd:int
sbc:succeeded-expects 1^^xsd:int
sbc:num-complains 1^^xsd:int
sbc:succeeded-expects 1^^xsd:int
sbc:content-item <urn:contentitem-{UUID}>
dc:created ... and other metadata
and the detailed results for Annotations
urn:result a sbc:BenchmarkResult
sbc:result-of urn:benchmarkExecution1
sbc:about urn:annotation1
sbc:state sbc:benchmark-state-succeeded (or sbc:benchmark-state-failed)
sbc:enhancement <urn:enhancement-{UUID}>, <urn:enhancement-{UUID}>
Note that a BenchmarkResult may refer to multiple enhancements as a expected/complained Entity could be suggested by several EntityAnnotation.
The Idea would be that the "BenchmarkExecution" provides most of the information needed for statistical analyses and the BenchmarkResults allow to directly go to the Enhancements and look into details.
Batch processing an Analyses could than be easily added - both on the client side and the server side.
WDYT
Rupert
> Benchmark should report evaluation summary
> ------------------------------------------
>
> Key: STANBOL-652
> URL: https://issues.apache.org/jira/browse/STANBOL-652
> Project: Stanbol
> Issue Type: Improvement
> Components: Testing
> Reporter: Pablo Mendes
> Priority: Minor
> Labels: benchmark, evaluation
>
> The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
> dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira