You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stanbol.apache.org by "Pablo Mendes (JIRA)" <ji...@apache.org> on 2012/06/04 12:49:25 UTC

[jira] [Created] (STANBOL-652) Benchmark should report evaluation summary

Pablo Mendes created STANBOL-652:
------------------------------------

             Summary: Benchmark should report evaluation summary
                 Key: STANBOL-652
                 URL: https://issues.apache.org/jira/browse/STANBOL-652
             Project: Stanbol
          Issue Type: Improvement
          Components: Testing
            Reporter: Pablo Mendes
            Priority: Minor


 The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
 dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (STANBOL-652) Benchmark should report evaluation summary

Posted by "Pablo Mendes (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/STANBOL-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401422#comment-13401422 ] 

Pablo Mendes commented on STANBOL-652:
--------------------------------------


I like it a lot. 

Saving the evaluation results to an RDF store would then allow one to slice-and-dice the results however he/she wants. 

For example, how many correct results for person annotations?

SELECT count(?result) 
WHERE {
  ?result sbc:state sbc:benchmark-state-succeeded .
  ?result sbc:about ?annotation .
  ?annotation entity ?person .
  ?person a dbpedia:Person .
}

Sharing this RDF via a SPARQL endpoint would then allow prompt Web-based report generation (aka visualization of results) with something like Sgvizler, for example 
http://code.google.com/p/sgvizler/

Although what I'd actually do would be to get a CSV from this and use R to analyze the results. But even getting this CSV should be trivial from the RDF.


                
> Benchmark should report evaluation summary
> ------------------------------------------
>
>                 Key: STANBOL-652
>                 URL: https://issues.apache.org/jira/browse/STANBOL-652
>             Project: Stanbol
>          Issue Type: Improvement
>          Components: Testing
>            Reporter: Pablo Mendes
>            Priority: Minor
>              Labels: benchmark, evaluation
>
>  The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
>  dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (STANBOL-652) Benchmark should report evaluation summary

Posted by "Rupert Westenthaler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/STANBOL-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401389#comment-13401389 ] 

Rupert Westenthaler commented on STANBOL-652:
---------------------------------------------

Hi Pablo

I think the best think to start would be to refactor the benchmarking module so that is can return Benchmark results as RDF

I would even suggest to convert the Benchmarks to RDF (but still supporting the Text based format currently used)

    urn:benchmark1 a sbc:Benchmark
        sbc:content "The INPUT text"
        sbc:expects urn:annotation1
        sbc:expects urn:annotation2
        sbc:complain urn:annotation3

    urn:annotation1 a sbc:Annotation
        entity <entity-URI>

NOTE: we might add different sbc:Annotation types in future (e.g. to validate Named Entities detection by NER tools or the detection of the correct language)


Results of a Benchmark execution

   urn:benchmarkExecution1 a BenchmarkExecution 
        sbc:benchmark urn:benchmark1
        sbc:num-expects 2^^xsd:int
        sbc:succeeded-expects 1^^xsd:int
        sbc:num-complains 1^^xsd:int
        sbc:succeeded-expects 1^^xsd:int
        sbc:content-item <urn:contentitem-{UUID}>
        dc:created ... and other metadata
        
and the detailed results for Annotations

    urn:result a sbc:BenchmarkResult
        sbc:result-of urn:benchmarkExecution1
        sbc:about urn:annotation1
        sbc:state sbc:benchmark-state-succeeded (or sbc:benchmark-state-failed)
        sbc:enhancement <urn:enhancement-{UUID}>, <urn:enhancement-{UUID}>     

Note that a BenchmarkResult may refer to multiple enhancements as a expected/complained Entity could be suggested by several EntityAnnotation.


The Idea would be that the "BenchmarkExecution" provides most of the information needed for statistical analyses and the BenchmarkResults allow to directly go to the Enhancements and look into details.

Batch processing an Analyses could than be easily added - both on the client side and the server side.

WDYT
Rupert
                
> Benchmark should report evaluation summary
> ------------------------------------------
>
>                 Key: STANBOL-652
>                 URL: https://issues.apache.org/jira/browse/STANBOL-652
>             Project: Stanbol
>          Issue Type: Improvement
>          Components: Testing
>            Reporter: Pablo Mendes
>            Priority: Minor
>              Labels: benchmark, evaluation
>
>  The SBC is a nice way to perform manual inspection of the behavior of the enhancement chain for different examples in the evaluation dataset. However, for evaluations with several hundreds of examples, it would be interesting to have scores that summarize the performance for the entire
>  dataset. For example, precision, recall and F1. An evaluation dataset is available here in BDL: http://spotlight.dbpedia.org/download/stanbol/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira