Lessons from a student appeal

[Ross Gardler <rg...@apache.org>]

In the 5 year history of GSoC we have only ever had two students appeal 
against a mid-term or full-term evaluation, one of which was this year.

Google take these appeals very seriously and the new Google lead on GSoC 
has the energy to follow up in some detail. So, here are the lessons 
learned from the examination of this appeal. I welcome any other 
suggestions that people might have for improving our process.

Summary
=======

Mentors and the project community must take ownership of the appeal and 
the decision, Apache admins cannot (and should not) police all mentor 
decisions. I'm pleased to report that this is exactly what happened in 
this case.

Apache admins should take responsibility for ensuring the process we 
follow is fair and balanced, that any appeal is taken seriously and that 
processes are modified if appropriate.

In this case I do not believe our process, the mentors or the project 
community are at fault. I do however suggest a minor tweak to help avoid 
these cases in the future.

Recommendation
-------------

In this case the lesson learned for our process is that we need to 
clearly communicate our expectations with respect to the students time 
commitments. We need to not only ask and evaluate "how much time will 
you put into GSoC" but also make it explicit in our communications with 
the student that putting in less time than this is likely to result in 
failure.

Action
------

Our mentee "roles and expectations" section of the documentation already 
included "The mentee is expected to dedicate a pre-agreed amount of time 
each week to the project", I have added "failure to
commit the agreed time is likely to result in failure on the programme".

These roles and responsibilities should be shared with all applicants to 
the mentoring programme via our GSoC documentation. They should also be 
sent to all successful students.

Detail
======

The student claimed that they had been interacting well with the 
community. This was confirmed by the mentor. However, the mentor and 
project community felt that progress made was not sufficient and there 
was a strong suspicion that the student was working as well as engaging 
with GSoC in their spare time.

Google engineers examined the code submitted by the student and although 
they did not object to the mentors decision they did wonder whether the 
project was of an achievable scope.

The mentor and project community responded with plenty of detail, 
including information about previous work that that paved the way for 
the student, that satisfied myself as an admin.

My position in this case
------------------------

Three years ago I asked Leslie Hawthorn about the significantly lower 
level of passes here in the ASF when compared to other organisations (if 
memory serves me right it is around 73% against 85%). I also asked our 
membership for their opinion.

The ASF membership felt that we might have higher standards here in the 
ASF.  Having met a great many mentors in other organisations I believe 
that (on average) this is probably true.

I won't repeat what Leslie said since it was a private conversation, 
however her points did not make me significantly alter my report back to 
the ASF.

The question then, and now, is whether or not our standards are too high.

I've been an admin since the very first year of GSoC (boy that was 
chaos). In all that time, I've been hands on with something like 250 
students. We've only ever had two appeals. One of which was a very clear 
case (very little effort). The other, this one, is less clear.

I would like to remind us all that the projects PMC feel that this 
student was trying to hold a day job down whilst also doing GSoC. We 
must remember that part of our evaluation is how many hours the student 
expects to spend on GSoC. In general if they are not close to full time 
they will not be selected.

With that in mind I see a small number of possible reasons for this failure:

a) The project was not reasonably scoped. In this case I do not believe 
this to be a problem and thus nothing needs changing in our process

b) ASF expectations are unreasonably high. I don't believe this to be 
the case (see discussion above). As an ex-computer science lecturer I'd 
say our expectations are about right, or even too low, since we are 
after the A grade students.

c) The student does not have the skills required to complete a 
reasonably scoped project - in which case our evaluation process is 
flawed. I don't think this is the case, see discussion above and 
remember our process has been refined and improved three times since 
those conversations.

d) the student did not have sufficient support from their mentor - this 
was not part of the students complaint (quite the opposite in fact) and 
thus not relevant in this case.

e) The student misled the evaluation team with respect to time available 
during the application process - in which case we need to not only ask 
"how much time will you put into GSoC" but also make it explicit that 
putting in less than this is likely to result in failure. The 
justification for this is that ASF expectations are high and therefore 
nothing but a full commitment to the project will result in success.

My conclusion is therefore that our standards should remain high. 
Lowering them reduces the value to our mentors and to future students. 
Instead we should (even more) clearly communicate our high expectations 
to applicants at the earliest possible stage.

Ross