How to score the model? (lift)

[Gustavo Frederico <gu...@thinkwrap.com>]

Considering PIO + the UR template, what are some of the ways to score
the model? I can think of some A/B testing but that would imply
relying on things outside the PIO + UR stack. Is there a way to
measure lift or "score the model"? Cross-validation with separate
testing set?

Thanks

Gustavo

Re: How to score the model? (lift)

[Gustavo Frederico <gu...@thinkwrap.com>]

I know of at least one good A/B testing service that uses JavaScript
at the Front End. So that would work and it wouldn't not touch the
PIO+UR stack. But your answer made me think of something else: Is
there a way to score two models with PIO?  Maybe Google Analytics is
still preferable even for that task...

Thanks
Gustavo

On Mon, Sep 26, 2016 at 7:21 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> A/B testing it the typical way to compare the performance of 2 or more versions of an app. You should make sure that the versions only vary in 1 way to isolate what you are comparing. So channel some traffic to a page “without recommendations” and some amount of traffic to a page “with recommendations” and measure your conversion KPI for both cases. If your conversion is a sale or purchase you may have to split all users permanently over the test period. if you are only measuring CTR you may be able to split impressions since you don’t have to track users as a group all the way to a purchase.
>
> On Sep 26, 2016, at 5:45 AM, Gustavo Frederico <gu...@thinkwrap.com> wrote:
>
> Allow me to elaborate some more: there's a second question related to
> the original one. How to score a model in Production?
>
> Thanks
>
> On Fri, Sep 23, 2016 at 2:58 PM, Gustavo Frederico
> <gu...@thinkwrap.com> wrote:
>> Considering PIO + the UR template, what are some of the ways to score
>> the model? I can think of some A/B testing but that would imply
>> relying on things outside the PIO + UR stack. Is there a way to
>> measure lift or "score the model"? Cross-validation with separate
>> testing set?
>>
>> Thanks
>>
>> Gustavo
>

Re: How to score the model? (lift)

[Pat Ferrel <pa...@occamsmachete.com>]

A/B testing it the typical way to compare the performance of 2 or more versions of an app. You should make sure that the versions only vary in 1 way to isolate what you are comparing. So channel some traffic to a page “without recommendations” and some amount of traffic to a page “with recommendations” and measure your conversion KPI for both cases. If your conversion is a sale or purchase you may have to split all users permanently over the test period. if you are only measuring CTR you may be able to split impressions since you don’t have to track users as a group all the way to a purchase.
  
On Sep 26, 2016, at 5:45 AM, Gustavo Frederico <gu...@thinkwrap.com> wrote:

Allow me to elaborate some more: there's a second question related to
the original one. How to score a model in Production?

Thanks

On Fri, Sep 23, 2016 at 2:58 PM, Gustavo Frederico
<gu...@thinkwrap.com> wrote:
> Considering PIO + the UR template, what are some of the ways to score
> the model? I can think of some A/B testing but that would imply
> relying on things outside the PIO + UR stack. Is there a way to
> measure lift or "score the model"? Cross-validation with separate
> testing set?
> 
> Thanks
> 
> Gustavo

Re: How to score the model? (lift)

[Gustavo Frederico <gu...@thinkwrap.com>]

Allow me to elaborate some more: there's a second question related to
the original one. How to score a model in Production?

Thanks

On Fri, Sep 23, 2016 at 2:58 PM, Gustavo Frederico
<gu...@thinkwrap.com> wrote:
> Considering PIO + the UR template, what are some of the ways to score
> the model? I can think of some A/B testing but that would imply
> relying on things outside the PIO + UR stack. Is there a way to
> measure lift or "score the model"? Cross-validation with separate
> testing set?
>
> Thanks
>
> Gustavo