Survey re-design kick-off

[Griselda Cuevas <gr...@apache.org>]

Hi Folks,

Here's the design doc[1] for the survey re-design project. This document is
what we will give to the vendor to scope the project.

Thanks!
G

[1]
https://docs.google.com/document/d/1mUNuHt6DFtVv2650_dPMsrrUwj7L0aGtDKBybWMLpw8/edit?usp=sharing

Re: Survey re-design kick-off

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> Well, one difference is that with software, we start with the expected
> result, a.k.a. requirements.

Not always, think of this more to an Agile development process to find out what we may need but we may not know it yet.

I think it’s reasonable to wait some progress have been made, and then provide some feedback or concerns if needed.

Any results we do eventually end up with should be able to be validated in a number of ways.

Thanks.
Justin

Re: Survey re-design kick-off

[Niclas Hedhman <ni...@hedhman.org>]

Well, one difference is that with software, we start with the expected
result, a.k.a. requirements.

On Wed, Jul 17, 2019 at 12:11 PM Ted Dunning <te...@gmail.com> wrote:

> The same sensitivity to small changes is true of software as well.
>
> Our normal response to that is to have experts involved in the design and
> be cautious about the results.
>
>
> On Tue, Jul 16, 2019 at 6:38 PM Niclas Hedhman <ni...@hedhman.org> wrote:
>
> > On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:
> >
> > > This is *not* a simple topic and just a single word or
> > > layout choice can change the results of your entire survey. Perilous!)
> > >
> >
> > And that doesn't sets off all the warning lights on the panel?? In my
> book,
> > a methodology that is that volatile shouldn't be used, and better
> > approaches should be searched for.
> >
> > // Niclas
> >
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Survey re-design kick-off

[Ted Dunning <te...@gmail.com>]

The same sensitivity to small changes is true of software as well.

Our normal response to that is to have experts involved in the design and
be cautious about the results.


On Tue, Jul 16, 2019 at 6:38 PM Niclas Hedhman <ni...@hedhman.org> wrote:

> On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:
>
> > This is *not* a simple topic and just a single word or
> > layout choice can change the results of your entire survey. Perilous!)
> >
>
> And that doesn't sets off all the warning lights on the panel?? In my book,
> a methodology that is that volatile shouldn't be used, and better
> approaches should be searched for.
>
> // Niclas
>

Re: Survey re-design kick-off

[Niclas Hedhman <ni...@hedhman.org>]

I am suggesting that when the design can strongly influence the outcome, it
would mean;
  a. the design could be (depending on the bias) be tailored for a
desirable outcome,
  b. the outcome not being reliable (and accepted), as one could always
criticize the design, one way or the other.

Niclas

On Wed, Jul 17, 2019 at 9:53 AM Joan Touzet <wo...@apache.org> wrote:

> Wait, you're suggesting throwing out *all* surveys because they're hard
> to use?
>
> All I was saying is that survey design is paramount and needs careful
> consideration, which is why we're contracting this out to a paid expert.
> It's a good decision, I think.
>
> On 2019-07-16 9:38 p.m., Niclas Hedhman wrote:
> > On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:
> >
> >> This is *not* a simple topic and just a single word or
> >> layout choice can change the results of your entire survey. Perilous!)
> >>
> >
> > And that doesn't sets off all the warning lights on the panel?? In my
> book,
> > a methodology that is that volatile shouldn't be used, and better
> > approaches should be searched for.
> >
> > // Niclas
> >
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Survey re-design kick-off

[Joan Touzet <wo...@apache.org>]

Wait, you're suggesting throwing out *all* surveys because they're hard 
to use?

All I was saying is that survey design is paramount and needs careful 
consideration, which is why we're contracting this out to a paid expert. 
It's a good decision, I think.

On 2019-07-16 9:38 p.m., Niclas Hedhman wrote:
> On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:
> 
>> This is *not* a simple topic and just a single word or
>> layout choice can change the results of your entire survey. Perilous!)
>>
> 
> And that doesn't sets off all the warning lights on the panel?? In my book,
> a methodology that is that volatile shouldn't be used, and better
> approaches should be searched for.
> 
> // Niclas
> 

Re: Survey re-design kick-off

[William A Rowe Jr <wr...@rowe-clan.net>]

On Tue, Jul 16, 2019, 20:38 Niclas Hedhman <ni...@hedhman.org> wrote:

> On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:
>
> > This is *not* a simple topic and just a single word or
> > layout choice can change the results of your entire survey. Perilous!)
> >
>
> And that doesn't sets off all the warning lights on the panel?? In my book,
> a methodology that is that volatile shouldn't be used, and better
> approaches should be searched for.
>
> // Niclas
>

Alarm bells? Of course, first you have lies, damned lies, and then you have
statistics.

More to the point, I administered thousands of hours of surveys to medical
and business professionals as a youth and was allowed no deviation or
clarifications (unless it was scripted for backup), all I could do was be
friendly, ask the stated question and record answers.

That's because the dozens of clients paid for accurate research designed by
the firm's educated and precise survey authors under contract by the
clients to get them accurate data they could use to run their
business/design their education campaign/understand the markets and
businesses within their umbrellas.

Professional guidance is very useful in market research. It keeps you from
lying to yourself and from being lied to.

Re: Survey re-design kick-off

[Niclas Hedhman <ni...@hedhman.org>]

On Wed, Jul 17, 2019 at 12:40 AM Joan Touzet <wo...@apache.org> wrote:

> This is *not* a simple topic and just a single word or
> layout choice can change the results of your entire survey. Perilous!)
>

And that doesn't sets off all the warning lights on the panel?? In my book,
a methodology that is that volatile shouldn't be used, and better
approaches should be searched for.

// Niclas

Re: Survey re-design kick-off

[Joan Touzet <wo...@apache.org>]

Resurrecting an old thread here because I think it deserves more attention.

Two resources I've skimmed but haven't dug into deeply:

https://stripe.com/en-de/atlas/guides/survey-design-principles

from Stripe. Obviously our consultant will be more skilled than this,
but if we're considering augmenting their work with our own, this is a
good read. (While working towards at PhD at OISE/University of Toronto,
survey design was something that was a doctorate-level research topic in
its own right. This is *not* a simple topic and just a single word or
layout choice can change the results of your entire survey. Perilous!)

https://github.com/chaoss/wg-diversity-inclusion

from CHAOSS, who we should be paying a lot more attention to. Some of
our ASF family, like Sharan Foga, participate in CHAOSS.

The proposed survey is closest to #2 and #7 in their metrics work, the
Contributor Community Diversity/Project and Cmmunity metrics. But there
a bunch of other areas CHAOSS focuses on we should consider looking into:

* Event Diversity
* Communication Inclusivity
* Recognition of Good Work
* Leadership
* Governance

-Joan

On 2019-07-12 9:53, Matt Sicker wrote:
> Doc looks interesting. The topics seem broad enough to get some interesting
> data, though I will note that I’m not experienced in surveying or anything
> like that.
> 
> On Fri, Jul 12, 2019 at 08:42, Niall Pemberton <ni...@gmail.com>
> wrote:
> 
>> On Fri, 12 Jul 2019 at 13:21, Myrle Krantz <my...@apache.org> wrote:
>>
>>> Hey all,
>>>
>>> This thread got lost in the storm the last few days.  Gris has, several
>>> times asked for feedback on things, and not gotten it (budget, values
>>> discussion, CoC, survey).  In the absence of feedback, y'all need to
>>> expect/accept that Gris will treat that as permission to do things the
>> way
>>> she has proposed.
>>
>>
>> Perhaps this is a stupid question, but does “demographic” include country
>> and language? Seems as a global org that would also be important.
>>
>> Niall
>>
>>
>>
>>>
>>> Best,
>>> Myrle
>>>
>>> On Fri, Jun 28, 2019 at 6:45 AM Griselda Cuevas <gr...@apache.org> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> Here's the design doc[1] for the survey re-design project. This
>> document
>>> is
>>>> what we will give to the vendor to scope the project.
>>>>
>>>> Thanks!
>>>> G
>>>>
>>>> [1]
>>>>
>>>>
>>>
>> https://docs.google.com/document/d/1mUNuHt6DFtVv2650_dPMsrrUwj7L0aGtDKBybWMLpw8/edit?usp=sharing
>>>>
>>>
>>

Re: Survey re-design kick-off

[Matt Sicker <bo...@gmail.com>]

Doc looks interesting. The topics seem broad enough to get some interesting
data, though I will note that I’m not experienced in surveying or anything
like that.

On Fri, Jul 12, 2019 at 08:42, Niall Pemberton <ni...@gmail.com>
wrote:

> On Fri, 12 Jul 2019 at 13:21, Myrle Krantz <my...@apache.org> wrote:
>
> > Hey all,
> >
> > This thread got lost in the storm the last few days.  Gris has, several
> > times asked for feedback on things, and not gotten it (budget, values
> > discussion, CoC, survey).  In the absence of feedback, y'all need to
> > expect/accept that Gris will treat that as permission to do things the
> way
> > she has proposed.
>
>
> Perhaps this is a stupid question, but does “demographic” include country
> and language? Seems as a global org that would also be important.
>
> Niall
>
>
>
> >
> > Best,
> > Myrle
> >
> > On Fri, Jun 28, 2019 at 6:45 AM Griselda Cuevas <gr...@apache.org> wrote:
> >
> > > Hi Folks,
> > >
> > > Here's the design doc[1] for the survey re-design project. This
> document
> > is
> > > what we will give to the vendor to scope the project.
> > >
> > > Thanks!
> > > G
> > >
> > > [1]
> > >
> > >
> >
> https://docs.google.com/document/d/1mUNuHt6DFtVv2650_dPMsrrUwj7L0aGtDKBybWMLpw8/edit?usp=sharing
> > >
> >
>
-- 
Matt Sicker <bo...@gmail.com>

Re: Survey re-design kick-off

[Niall Pemberton <ni...@gmail.com>]

On Fri, 12 Jul 2019 at 13:21, Myrle Krantz <my...@apache.org> wrote:

> Hey all,
>
> This thread got lost in the storm the last few days.  Gris has, several
> times asked for feedback on things, and not gotten it (budget, values
> discussion, CoC, survey).  In the absence of feedback, y'all need to
> expect/accept that Gris will treat that as permission to do things the way
> she has proposed.


Perhaps this is a stupid question, but does “demographic” include country
and language? Seems as a global org that would also be important.

Niall



>
> Best,
> Myrle
>
> On Fri, Jun 28, 2019 at 6:45 AM Griselda Cuevas <gr...@apache.org> wrote:
>
> > Hi Folks,
> >
> > Here's the design doc[1] for the survey re-design project. This document
> is
> > what we will give to the vendor to scope the project.
> >
> > Thanks!
> > G
> >
> > [1]
> >
> >
> https://docs.google.com/document/d/1mUNuHt6DFtVv2650_dPMsrrUwj7L0aGtDKBybWMLpw8/edit?usp=sharing
> >
>

Re: Survey re-design kick-off

[Myrle Krantz <my...@apache.org>]

Hey all,

This thread got lost in the storm the last few days.  Gris has, several
times asked for feedback on things, and not gotten it (budget, values
discussion, CoC, survey).  In the absence of feedback, y'all need to
expect/accept that Gris will treat that as permission to do things the way
she has proposed.

Best,
Myrle

On Fri, Jun 28, 2019 at 6:45 AM Griselda Cuevas <gr...@apache.org> wrote:

> Hi Folks,
>
> Here's the design doc[1] for the survey re-design project. This document is
> what we will give to the vendor to scope the project.
>
> Thanks!
> G
>
> [1]
>
> https://docs.google.com/document/d/1mUNuHt6DFtVv2650_dPMsrrUwj7L0aGtDKBybWMLpw8/edit?usp=sharing
>