You are viewing a plain text version of this content. The canonical link for it is here.
Posted to openrelevance-user@lucene.apache.org by Mark Bennett <mb...@ideaeng.com> on 2010/02/11 18:58:48 UTC
Comments on ORP Wiki Additions ?
Good morning Relevancy comrades,
I've tried to take a stab at outlining this rather complex subject in the
wiki. Of course it's a work in progress.
I've done a high level outline here:
http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
And an expansion of the first section of the outline here:
http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
I actually could use some feedback. I promise you this is not vanity, there
are actually some very pragmatic motives for my postings.
I guess some specific questions:
* I'm trying to create a bit of a "crash course" in Relevancy Testing, are
there major areas I've overlooked?
* I've outlined 2 broad categories of testing, do you agree?
* I've tried to explore some of the high level strengths and drawbacks of
certain methodologies
* Is the "tone" reasonably neutral? What I mean is that some folks may be
attached to certain methods, I don't want to seem like I'm "trashing"
anything, just trying to point out the strengths and weaknesses in a fair
way.
I look forward to any comments.
Mark
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
Re: Comments on ORP Wiki Additions ?
Posted by "Avi Rappoport, Search Tools Consulting" <co...@searchtools.com>.
Resources for Multilingual Search (aka CLIR, Cross-lingual
Information Retrieval):
Information Access in a Multilingual World (SIGIR 2009)
a workshop (still looking for proceedings)
http://www.sics.se/events/clir2009
and one report (I think it will be in the SigIR Forum, maybe June)
http://mitcho.com/blog/projects/report-from-sigir-workshop-on-information-access-in-a-multilingual-world/
UC Berkeley is working on Multilingual Entry Vocabulary
http://metadata.berkeley.edu/prototypesI.html
and multilingual geotemporal information retrieval
http://metadata.berkeley.edu/NTCIR-GeoTime/description.php
Fred Gey is The Man
http://ucdata.berkeley.edu/gey.html
At 9:58 AM -0800 2/11/10, Mark Bennett wrote:
>Good morning Relevancy comrades,
>
>I've tried to take a stab at outlining this rather complex subject
>in the wiki. Of course it's a work in progress.
>
>I've done a high level outline here:
><http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline>http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>
>And an expansion of the first section of the outline here:
><http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing>http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>
>I actually could use some feedback. I promise you this is not
>vanity, there are actually some very pragmatic motives for my
>postings.
>
>I guess some specific questions:
>* I'm trying to create a bit of a "crash course" in Relevancy
>Testing, are there major areas I've overlooked?
>* I've outlined 2 broad categories of testing, do you agree?
>* I've tried to explore some of the high level strengths and
>drawbacks of certain methodologies
>* Is the "tone" reasonably neutral? What I mean is that some folks
>may be attached to certain methods, I don't want to seem like I'm
>"trashing" anything, just trying to point out the strengths and
>weaknesses in a fair way.
>
>I look forward to any comments.
>
>Mark
>
>--
>Mark Bennett / New Idea Engineering, Inc. /
><ma...@ideaeng.com>mbennett@ideaeng.com
>Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
--
Search Tools Consulting: expert advice for Enterprise Search
(510) 845-2551 -- <mailto: consult9@searchtools.com>
SearchTools: Guide to Search Engines for Web Sites and Intranets
<http://www.searchtools.com>
Re: Comments on ORP Wiki Additions ?
Posted by Robert Muir <rc...@gmail.com>.
I am sure there are better (much older) papers to describe pooling that
might help in addition.
that paper is more from an or-user perspective, a guide to getting more
realistic use out of a test collection.
we should separately look for some good stuff from an or-dev perspective if
we want to properly create pooled judgements for a test collection, its a
little strange, who would be in the pool? (its not like a trec conference
where you have a finite set of teams and they are all getting pooled
equally)
On Thu, Feb 11, 2010 at 7:42 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> Robert,
>
> That link was awesome, thank you! I've added it to the detailed page.
>
> Also, I've taken a stab at an Introduction on the outline page. Oddly,
> Confluence seems to need manual refreshing more than other wikis I use, even
> days later. I wonder if there's a cache setting or something...
>
> With regards to outline, in some places it's perhaps more terse than
> technical. In other projects I've found that even an incomplete outline can
> evolve into a great resource.
>
> With regards to code, I've been working on some stuff that interacts with
> multiple search engines in their native format and translates into a common
> Atom feed, along the lines of the OpenSearch format. This is in the "you
> want it, you build it". Our interest in ORP is very cross-engine centric.
>
> Still lots of details to work through. If anybody knows XSLT *really* well
> I'd like to bend their ear, having some issues with namespaces.
>
>
> --
> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>
>
> On Thu, Feb 11, 2010 at 1:42 PM, Robert Muir <rc...@gmail.com> wrote:
>
>> only a partial subset of the docs (some top-N from different submissions)
>> are placed into a pool and judged.
>>
>> here is a great little presentation that is very relevant to ORP project,
>> as i am sure we don't want to create complete judgements, yet we want
>> reusable evaluation collections:
>> http://www.ir.uwaterloo.ca/slides/buettcher_reliable_evaluation.pdf
>>
>>
>> On Thu, Feb 11, 2010 at 4:31 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>
>>> Hi Robert,
>>>
>>> By "pooling", you mean they combine different sets of source docs and
>>> question sets, in kind of a patch work? If that's what you mean, do you
>>> know how that process was generally done? How close to "perfection", ie
>>> total coverage by humans, do you think they got?
>>>
>>> If that's not what you meant by "pooling" then I'm a bit confused...
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>> --
>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>
>>>
>>> On Thu, Feb 11, 2010 at 1:02 PM, Robert Muir <rc...@gmail.com> wrote:
>>>
>>>> in this case pooling is what is typically used.
>>>>
>>>>
>>>> On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>>
>>>>> Thanks Robert,
>>>>>
>>>>> Excellent comments, I'll try to add something to the outline. Either a
>>>>> higher level top section, or some intro text.
>>>>>
>>>>> Robert, in particular, I wonder if you could look at:
>>>>>
>>>>>
>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>>
>>>>> In the section on "Full-Grid Assertions (TREC-Style!)"
>>>>>
>>>>> It talks about the "M x N" problem of creating relevancy judgment
>>>>> data. It also explores some of the shortcuts that could be used.
>>>>>
>>>>> We're actually working through these problems with a couple clients.
>>>>> On the one hand they want "perfect" measurements, but on the other hand
>>>>> nobody wants to fund the work to create completely curated test sets. This
>>>>> is the classic "good vs. cheap" argument, and I DO think there are
>>>>> reasonable compromises to be had.
>>>>>
>>>>> TREC has evolved over the years and I wonder how they've addressed
>>>>> these. Did they take any shortcuts? Or did they get enough manpower to
>>>>> really curate every single document and relevancy judgment?
>>>>>
>>>>> I'll be adding more about some of the compromises we've considered and
>>>>> worked on, but it'd be great to get other experts to chime in. Either y'all
>>>>> will come back with other ideas we didn't think, or we get to say "we told
>>>>> you so" - I'm happy either way.
>>>>>
>>>>> And what I love about the ORP process is that all of this is captured
>>>>> and vetted in an accessible public forum. TREC was also peer reviewed, so
>>>>> this continues that tradition in the newer medium. And I'll work on an even
>>>>> clearer outline
>>>>>
>>>>>
>>>>> Mark
>>>>>
>>>>> --
>>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>>
>>>>>
>>>>> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com>wrote:
>>>>>
>>>>>> first of all, thanks for adding this content!
>>>>>>
>>>>>> in my opinion one thing that might be helpful would be an
>>>>>> 'introduction' section that is VERY high-level. I don't want to sound
>>>>>> negative but your 'high level outline' is actually quite technical :)
>>>>>>
>>>>>> it might be a good thing for this project if we had some content
>>>>>> somewhere that explained at a very very high level what this whole relevance
>>>>>> testing thing is all about...
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>>>>
>>>>>>> Good morning Relevancy comrades,
>>>>>>>
>>>>>>> I've tried to take a stab at outlining this rather complex subject in
>>>>>>> the wiki. Of course it's a work in progress.
>>>>>>>
>>>>>>> I've done a high level outline here:
>>>>>>>
>>>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>>>>>>
>>>>>>> And an expansion of the first section of the outline here:
>>>>>>>
>>>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>>>>
>>>>>>> I actually could use some feedback. I promise you this is not
>>>>>>> vanity, there are actually some very pragmatic motives for my postings.
>>>>>>>
>>>>>>> I guess some specific questions:
>>>>>>> * I'm trying to create a bit of a "crash course" in Relevancy
>>>>>>> Testing, are there major areas I've overlooked?
>>>>>>> * I've outlined 2 broad categories of testing, do you agree?
>>>>>>> * I've tried to explore some of the high level strengths and
>>>>>>> drawbacks of certain methodologies
>>>>>>> * Is the "tone" reasonably neutral? What I mean is that some folks
>>>>>>> may be attached to certain methods, I don't want to seem like I'm "trashing"
>>>>>>> anything, just trying to point out the strengths and weaknesses in a fair
>>>>>>> way.
>>>>>>>
>>>>>>> I look forward to any comments.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> --
>>>>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Robert Muir
>>>>>> rcmuir@gmail.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Robert Muir
>>>> rcmuir@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Comments on ORP Wiki Additions ?
Posted by Mark Bennett <mb...@ideaeng.com>.
Robert,
That link was awesome, thank you! I've added it to the detailed page.
Also, I've taken a stab at an Introduction on the outline page. Oddly,
Confluence seems to need manual refreshing more than other wikis I use, even
days later. I wonder if there's a cache setting or something...
With regards to outline, in some places it's perhaps more terse than
technical. In other projects I've found that even an incomplete outline can
evolve into a great resource.
With regards to code, I've been working on some stuff that interacts with
multiple search engines in their native format and translates into a common
Atom feed, along the lines of the OpenSearch format. This is in the "you
want it, you build it". Our interest in ORP is very cross-engine centric.
Still lots of details to work through. If anybody knows XSLT *really* well
I'd like to bend their ear, having some issues with namespaces.
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
On Thu, Feb 11, 2010 at 1:42 PM, Robert Muir <rc...@gmail.com> wrote:
> only a partial subset of the docs (some top-N from different submissions)
> are placed into a pool and judged.
>
> here is a great little presentation that is very relevant to ORP project,
> as i am sure we don't want to create complete judgements, yet we want
> reusable evaluation collections:
> http://www.ir.uwaterloo.ca/slides/buettcher_reliable_evaluation.pdf
>
>
> On Thu, Feb 11, 2010 at 4:31 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>
>> Hi Robert,
>>
>> By "pooling", you mean they combine different sets of source docs and
>> question sets, in kind of a patch work? If that's what you mean, do you
>> know how that process was generally done? How close to "perfection", ie
>> total coverage by humans, do you think they got?
>>
>> If that's not what you meant by "pooling" then I'm a bit confused...
>>
>> Thanks,
>>
>> Mark
>>
>> --
>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>
>>
>> On Thu, Feb 11, 2010 at 1:02 PM, Robert Muir <rc...@gmail.com> wrote:
>>
>>> in this case pooling is what is typically used.
>>>
>>>
>>> On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>
>>>> Thanks Robert,
>>>>
>>>> Excellent comments, I'll try to add something to the outline. Either a
>>>> higher level top section, or some intro text.
>>>>
>>>> Robert, in particular, I wonder if you could look at:
>>>>
>>>>
>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>
>>>> In the section on "Full-Grid Assertions (TREC-Style!)"
>>>>
>>>> It talks about the "M x N" problem of creating relevancy judgment data.
>>>> It also explores some of the shortcuts that could be used.
>>>>
>>>> We're actually working through these problems with a couple clients. On
>>>> the one hand they want "perfect" measurements, but on the other hand nobody
>>>> wants to fund the work to create completely curated test sets. This is the
>>>> classic "good vs. cheap" argument, and I DO think there are reasonable
>>>> compromises to be had.
>>>>
>>>> TREC has evolved over the years and I wonder how they've addressed
>>>> these. Did they take any shortcuts? Or did they get enough manpower to
>>>> really curate every single document and relevancy judgment?
>>>>
>>>> I'll be adding more about some of the compromises we've considered and
>>>> worked on, but it'd be great to get other experts to chime in. Either y'all
>>>> will come back with other ideas we didn't think, or we get to say "we told
>>>> you so" - I'm happy either way.
>>>>
>>>> And what I love about the ORP process is that all of this is captured
>>>> and vetted in an accessible public forum. TREC was also peer reviewed, so
>>>> this continues that tradition in the newer medium. And I'll work on an even
>>>> clearer outline
>>>>
>>>>
>>>> Mark
>>>>
>>>> --
>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>
>>>>
>>>> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com> wrote:
>>>>
>>>>> first of all, thanks for adding this content!
>>>>>
>>>>> in my opinion one thing that might be helpful would be an
>>>>> 'introduction' section that is VERY high-level. I don't want to sound
>>>>> negative but your 'high level outline' is actually quite technical :)
>>>>>
>>>>> it might be a good thing for this project if we had some content
>>>>> somewhere that explained at a very very high level what this whole relevance
>>>>> testing thing is all about...
>>>>>
>>>>>
>>>>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>>>
>>>>>> Good morning Relevancy comrades,
>>>>>>
>>>>>> I've tried to take a stab at outlining this rather complex subject in
>>>>>> the wiki. Of course it's a work in progress.
>>>>>>
>>>>>> I've done a high level outline here:
>>>>>>
>>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>>>>>
>>>>>> And an expansion of the first section of the outline here:
>>>>>>
>>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>>>
>>>>>> I actually could use some feedback. I promise you this is not vanity,
>>>>>> there are actually some very pragmatic motives for my postings.
>>>>>>
>>>>>> I guess some specific questions:
>>>>>> * I'm trying to create a bit of a "crash course" in Relevancy Testing,
>>>>>> are there major areas I've overlooked?
>>>>>> * I've outlined 2 broad categories of testing, do you agree?
>>>>>> * I've tried to explore some of the high level strengths and drawbacks
>>>>>> of certain methodologies
>>>>>> * Is the "tone" reasonably neutral? What I mean is that some folks
>>>>>> may be attached to certain methods, I don't want to seem like I'm "trashing"
>>>>>> anything, just trying to point out the strengths and weaknesses in a fair
>>>>>> way.
>>>>>>
>>>>>> I look forward to any comments.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> --
>>>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Robert Muir
>>>>> rcmuir@gmail.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Robert Muir
>>> rcmuir@gmail.com
>>>
>>
>>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
Re: Comments on ORP Wiki Additions ?
Posted by Robert Muir <rc...@gmail.com>.
only a partial subset of the docs (some top-N from different submissions)
are placed into a pool and judged.
here is a great little presentation that is very relevant to ORP project, as
i am sure we don't want to create complete judgements, yet we want reusable
evaluation collections:
http://www.ir.uwaterloo.ca/slides/buettcher_reliable_evaluation.pdf
On Thu, Feb 11, 2010 at 4:31 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> Hi Robert,
>
> By "pooling", you mean they combine different sets of source docs and
> question sets, in kind of a patch work? If that's what you mean, do you
> know how that process was generally done? How close to "perfection", ie
> total coverage by humans, do you think they got?
>
> If that's not what you meant by "pooling" then I'm a bit confused...
>
> Thanks,
>
> Mark
>
> --
> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>
>
> On Thu, Feb 11, 2010 at 1:02 PM, Robert Muir <rc...@gmail.com> wrote:
>
>> in this case pooling is what is typically used.
>>
>>
>> On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>
>>> Thanks Robert,
>>>
>>> Excellent comments, I'll try to add something to the outline. Either a
>>> higher level top section, or some intro text.
>>>
>>> Robert, in particular, I wonder if you could look at:
>>>
>>>
>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>
>>> In the section on "Full-Grid Assertions (TREC-Style!)"
>>>
>>> It talks about the "M x N" problem of creating relevancy judgment data.
>>> It also explores some of the shortcuts that could be used.
>>>
>>> We're actually working through these problems with a couple clients. On
>>> the one hand they want "perfect" measurements, but on the other hand nobody
>>> wants to fund the work to create completely curated test sets. This is the
>>> classic "good vs. cheap" argument, and I DO think there are reasonable
>>> compromises to be had.
>>>
>>> TREC has evolved over the years and I wonder how they've addressed
>>> these. Did they take any shortcuts? Or did they get enough manpower to
>>> really curate every single document and relevancy judgment?
>>>
>>> I'll be adding more about some of the compromises we've considered and
>>> worked on, but it'd be great to get other experts to chime in. Either y'all
>>> will come back with other ideas we didn't think, or we get to say "we told
>>> you so" - I'm happy either way.
>>>
>>> And what I love about the ORP process is that all of this is captured and
>>> vetted in an accessible public forum. TREC was also peer reviewed, so this
>>> continues that tradition in the newer medium. And I'll work on an even
>>> clearer outline
>>>
>>>
>>> Mark
>>>
>>> --
>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>
>>>
>>> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com> wrote:
>>>
>>>> first of all, thanks for adding this content!
>>>>
>>>> in my opinion one thing that might be helpful would be an 'introduction'
>>>> section that is VERY high-level. I don't want to sound negative but your
>>>> 'high level outline' is actually quite technical :)
>>>>
>>>> it might be a good thing for this project if we had some content
>>>> somewhere that explained at a very very high level what this whole relevance
>>>> testing thing is all about...
>>>>
>>>>
>>>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>>
>>>>> Good morning Relevancy comrades,
>>>>>
>>>>> I've tried to take a stab at outlining this rather complex subject in
>>>>> the wiki. Of course it's a work in progress.
>>>>>
>>>>> I've done a high level outline here:
>>>>>
>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>>>>
>>>>> And an expansion of the first section of the outline here:
>>>>>
>>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>>
>>>>> I actually could use some feedback. I promise you this is not vanity,
>>>>> there are actually some very pragmatic motives for my postings.
>>>>>
>>>>> I guess some specific questions:
>>>>> * I'm trying to create a bit of a "crash course" in Relevancy Testing,
>>>>> are there major areas I've overlooked?
>>>>> * I've outlined 2 broad categories of testing, do you agree?
>>>>> * I've tried to explore some of the high level strengths and drawbacks
>>>>> of certain methodologies
>>>>> * Is the "tone" reasonably neutral? What I mean is that some folks may
>>>>> be attached to certain methods, I don't want to seem like I'm "trashing"
>>>>> anything, just trying to point out the strengths and weaknesses in a fair
>>>>> way.
>>>>>
>>>>> I look forward to any comments.
>>>>>
>>>>> Mark
>>>>>
>>>>> --
>>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Robert Muir
>>>> rcmuir@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Comments on ORP Wiki Additions ?
Posted by Mark Bennett <mb...@ideaeng.com>.
Hi Robert,
By "pooling", you mean they combine different sets of source docs and
question sets, in kind of a patch work? If that's what you mean, do you
know how that process was generally done? How close to "perfection", ie
total coverage by humans, do you think they got?
If that's not what you meant by "pooling" then I'm a bit confused...
Thanks,
Mark
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
On Thu, Feb 11, 2010 at 1:02 PM, Robert Muir <rc...@gmail.com> wrote:
> in this case pooling is what is typically used.
>
>
> On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>
>> Thanks Robert,
>>
>> Excellent comments, I'll try to add something to the outline. Either a
>> higher level top section, or some intro text.
>>
>> Robert, in particular, I wonder if you could look at:
>>
>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>
>> In the section on "Full-Grid Assertions (TREC-Style!)"
>>
>> It talks about the "M x N" problem of creating relevancy judgment data.
>> It also explores some of the shortcuts that could be used.
>>
>> We're actually working through these problems with a couple clients. On
>> the one hand they want "perfect" measurements, but on the other hand nobody
>> wants to fund the work to create completely curated test sets. This is the
>> classic "good vs. cheap" argument, and I DO think there are reasonable
>> compromises to be had.
>>
>> TREC has evolved over the years and I wonder how they've addressed these.
>> Did they take any shortcuts? Or did they get enough manpower to really
>> curate every single document and relevancy judgment?
>>
>> I'll be adding more about some of the compromises we've considered and
>> worked on, but it'd be great to get other experts to chime in. Either y'all
>> will come back with other ideas we didn't think, or we get to say "we told
>> you so" - I'm happy either way.
>>
>> And what I love about the ORP process is that all of this is captured and
>> vetted in an accessible public forum. TREC was also peer reviewed, so this
>> continues that tradition in the newer medium. And I'll work on an even
>> clearer outline
>>
>>
>> Mark
>>
>> --
>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>
>>
>> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com> wrote:
>>
>>> first of all, thanks for adding this content!
>>>
>>> in my opinion one thing that might be helpful would be an 'introduction'
>>> section that is VERY high-level. I don't want to sound negative but your
>>> 'high level outline' is actually quite technical :)
>>>
>>> it might be a good thing for this project if we had some content
>>> somewhere that explained at a very very high level what this whole relevance
>>> testing thing is all about...
>>>
>>>
>>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>>
>>>> Good morning Relevancy comrades,
>>>>
>>>> I've tried to take a stab at outlining this rather complex subject in
>>>> the wiki. Of course it's a work in progress.
>>>>
>>>> I've done a high level outline here:
>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>>>
>>>> And an expansion of the first section of the outline here:
>>>>
>>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>>
>>>> I actually could use some feedback. I promise you this is not vanity,
>>>> there are actually some very pragmatic motives for my postings.
>>>>
>>>> I guess some specific questions:
>>>> * I'm trying to create a bit of a "crash course" in Relevancy Testing,
>>>> are there major areas I've overlooked?
>>>> * I've outlined 2 broad categories of testing, do you agree?
>>>> * I've tried to explore some of the high level strengths and drawbacks
>>>> of certain methodologies
>>>> * Is the "tone" reasonably neutral? What I mean is that some folks may
>>>> be attached to certain methods, I don't want to seem like I'm "trashing"
>>>> anything, just trying to point out the strengths and weaknesses in a fair
>>>> way.
>>>>
>>>> I look forward to any comments.
>>>>
>>>> Mark
>>>>
>>>> --
>>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>>
>>>
>>>
>>>
>>> --
>>> Robert Muir
>>> rcmuir@gmail.com
>>>
>>
>>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
Re: Comments on ORP Wiki Additions ?
Posted by Robert Muir <rc...@gmail.com>.
in this case pooling is what is typically used.
On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> Thanks Robert,
>
> Excellent comments, I'll try to add something to the outline. Either a
> higher level top section, or some intro text.
>
> Robert, in particular, I wonder if you could look at:
>
> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>
> In the section on "Full-Grid Assertions (TREC-Style!)"
>
> It talks about the "M x N" problem of creating relevancy judgment data. It
> also explores some of the shortcuts that could be used.
>
> We're actually working through these problems with a couple clients. On
> the one hand they want "perfect" measurements, but on the other hand nobody
> wants to fund the work to create completely curated test sets. This is the
> classic "good vs. cheap" argument, and I DO think there are reasonable
> compromises to be had.
>
> TREC has evolved over the years and I wonder how they've addressed these.
> Did they take any shortcuts? Or did they get enough manpower to really
> curate every single document and relevancy judgment?
>
> I'll be adding more about some of the compromises we've considered and
> worked on, but it'd be great to get other experts to chime in. Either y'all
> will come back with other ideas we didn't think, or we get to say "we told
> you so" - I'm happy either way.
>
> And what I love about the ORP process is that all of this is captured and
> vetted in an accessible public forum. TREC was also peer reviewed, so this
> continues that tradition in the newer medium. And I'll work on an even
> clearer outline
>
>
> Mark
>
> --
> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>
>
> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com> wrote:
>
>> first of all, thanks for adding this content!
>>
>> in my opinion one thing that might be helpful would be an 'introduction'
>> section that is VERY high-level. I don't want to sound negative but your
>> 'high level outline' is actually quite technical :)
>>
>> it might be a good thing for this project if we had some content somewhere
>> that explained at a very very high level what this whole relevance testing
>> thing is all about...
>>
>>
>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>>
>>> Good morning Relevancy comrades,
>>>
>>> I've tried to take a stab at outlining this rather complex subject in the
>>> wiki. Of course it's a work in progress.
>>>
>>> I've done a high level outline here:
>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>>
>>> And an expansion of the first section of the outline here:
>>>
>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>>
>>> I actually could use some feedback. I promise you this is not vanity,
>>> there are actually some very pragmatic motives for my postings.
>>>
>>> I guess some specific questions:
>>> * I'm trying to create a bit of a "crash course" in Relevancy Testing,
>>> are there major areas I've overlooked?
>>> * I've outlined 2 broad categories of testing, do you agree?
>>> * I've tried to explore some of the high level strengths and drawbacks of
>>> certain methodologies
>>> * Is the "tone" reasonably neutral? What I mean is that some folks may
>>> be attached to certain methods, I don't want to seem like I'm "trashing"
>>> anything, just trying to point out the strengths and weaknesses in a fair
>>> way.
>>>
>>> I look forward to any comments.
>>>
>>> Mark
>>>
>>> --
>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>>
>>
>>
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Comments on ORP Wiki Additions ?
Posted by Mark Bennett <mb...@ideaeng.com>.
Thanks Robert,
Excellent comments, I'll try to add something to the outline. Either a
higher level top section, or some intro text.
Robert, in particular, I wonder if you could look at:
http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
In the section on "Full-Grid Assertions (TREC-Style!)"
It talks about the "M x N" problem of creating relevancy judgment data. It
also explores some of the shortcuts that could be used.
We're actually working through these problems with a couple clients. On the
one hand they want "perfect" measurements, but on the other hand nobody
wants to fund the work to create completely curated test sets. This is the
classic "good vs. cheap" argument, and I DO think there are reasonable
compromises to be had.
TREC has evolved over the years and I wonder how they've addressed these.
Did they take any shortcuts? Or did they get enough manpower to really
curate every single document and relevancy judgment?
I'll be adding more about some of the compromises we've considered and
worked on, but it'd be great to get other experts to chime in. Either y'all
will come back with other ideas we didn't think, or we get to say "we told
you so" - I'm happy either way.
And what I love about the ORP process is that all of this is captured and
vetted in an accessible public forum. TREC was also peer reviewed, so this
continues that tradition in the newer medium. And I'll work on an even
clearer outline
Mark
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir <rc...@gmail.com> wrote:
> first of all, thanks for adding this content!
>
> in my opinion one thing that might be helpful would be an 'introduction'
> section that is VERY high-level. I don't want to sound negative but your
> 'high level outline' is actually quite technical :)
>
> it might be a good thing for this project if we had some content somewhere
> that explained at a very very high level what this whole relevance testing
> thing is all about...
>
>
> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com>wrote:
>
>> Good morning Relevancy comrades,
>>
>> I've tried to take a stab at outlining this rather complex subject in the
>> wiki. Of course it's a work in progress.
>>
>> I've done a high level outline here:
>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>>
>> And an expansion of the first section of the outline here:
>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>>
>> I actually could use some feedback. I promise you this is not vanity,
>> there are actually some very pragmatic motives for my postings.
>>
>> I guess some specific questions:
>> * I'm trying to create a bit of a "crash course" in Relevancy Testing, are
>> there major areas I've overlooked?
>> * I've outlined 2 broad categories of testing, do you agree?
>> * I've tried to explore some of the high level strengths and drawbacks of
>> certain methodologies
>> * Is the "tone" reasonably neutral? What I mean is that some folks may be
>> attached to certain methods, I don't want to seem like I'm "trashing"
>> anything, just trying to point out the strengths and weaknesses in a fair
>> way.
>>
>> I look forward to any comments.
>>
>> Mark
>>
>> --
>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>>
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>
Re: Comments on ORP Wiki Additions ?
Posted by Robert Muir <rc...@gmail.com>.
first of all, thanks for adding this content!
in my opinion one thing that might be helpful would be an 'introduction'
section that is VERY high-level. I don't want to sound negative but your
'high level outline' is actually quite technical :)
it might be a good thing for this project if we had some content somewhere
that explained at a very very high level what this whole relevance testing
thing is all about...
On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> Good morning Relevancy comrades,
>
> I've tried to take a stab at outlining this rather complex subject in the
> wiki. Of course it's a work in progress.
>
> I've done a high level outline here:
> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline
>
> And an expansion of the first section of the outline here:
> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing
>
> I actually could use some feedback. I promise you this is not vanity,
> there are actually some very pragmatic motives for my postings.
>
> I guess some specific questions:
> * I'm trying to create a bit of a "crash course" in Relevancy Testing, are
> there major areas I've overlooked?
> * I've outlined 2 broad categories of testing, do you agree?
> * I've tried to explore some of the high level strengths and drawbacks of
> certain methodologies
> * Is the "tone" reasonably neutral? What I mean is that some folks may be
> attached to certain methods, I don't want to seem like I'm "trashing"
> anything, just trying to point out the strengths and weaknesses in a fair
> way.
>
> I look forward to any comments.
>
> Mark
>
> --
> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
>
--
Robert Muir
rcmuir@gmail.com