AW: Usability evaluation report

[Matthias Niederhausen <ma...@t-systems-mms.com>]

Hello Stan,

sorry for getting back so late, but I was ensnared in yet another user study
(with some different focus) in China for the last few weeks. Hopefully, I
will be able to post some results we got as a "side-effect" of this new
study soon.
I will try to address the issues you mentioned, also considering our latest
findings:

1) In the Omelette project, we have created several means to add widgets
directly to a workspace using recommendations. While the quality of
recommendations is a tricky topic, users like the ability to add new widgets
without leaving their workspace. One tractable option for this might be a
search field on the top of the page offering a direct result list so I can
click a widget to add it to the workspace. I am inspired by the twitter
search field here, which offers icons and additional data on the result
(e.g., type of result) immediately.
On the other hand, virtually all of our users did not expect the page layout
to be restricted to some fixed number of columns (that has to be edited
separately). Nearly all of them tried to instinctively drag&drop widgets
like icons on a desktop or views in Eclipse and became frustrated when they
could not do so. So, the default layout of a page should be something that
dynamically allocates space (maybe using a dynamic number of columns).

2) I am very glad to see that you agree on the feedback usability issues.
One particular idea that I have had is that upon opening a workspace, the
widget frames should be overlayed by the big widget icon until they have
loaded and some "loading" indicator. This way, users will immediately know
that their browser has not crashed and where which widget is.

3) Thank you for your concerns regarding the formalties of the report. This
was the first usability report I have ever created and I will be glad to
improve the structure of future reports with your hints. I have now heard of
Techsmith Morae for the first time and it looks very promising. In our
studies, however, we have also used "discount" approaches (Google Docs
questionaires, measuring times by stopwatch, recording screen, audio and
camera using the tool XSplit).


All the best,
Matthias


-----Ursprüngliche Nachricht-----
Von: Drozdetski, Stan A. [mailto:drozdetski@mitre.org] 
Gesendet: Dienstag, 13. November 2012 23:15
An: dev@rave.apache.org
Cc: Matthias Niederhausen
Betreff: RE: Usability evaluation report

Matthias, thank you for sharing this with the community! Very helpful to see
not just the summary of findings, but also your entire protocol.

I'm picking out two major issue groups in need of improvement:

1) Navigation to/from widget store + workflow for adding gadgets In our
internal testing, we've seen quite a wide variety of mental models that
people approach gadget-based interfaces with. Even folks with experience in
other portal systems may not instinctively know which parts of the page are
modifiable, and how to adjust the page contents vs page layout. We've tried
video tutorials and first-time-use hints, but can't quite claim a silver
bullet just yet.

Another challenge is the separation between the widget store and the actual
layout; it simply puts a barrier in the way of direct interface
manipulation. I've been thinking through some solutions that show the user
what's available without taking them off the page, but screen real estate
becomes an issue. If you or anyone else have seen innovative solutions,
please do share with the list. 

2) Feedback for actions
Yes, no feedback (or feedback that the user can't see) is a usability death
spell. Same goes for long loading time. Thank  you for noting that, and I
agree with your recommendations 100%.

Couple of unsolicited thoughts on the report itself - in case you're
interested in what worked/didn't work for us at MITRE, or if there are other
usability nerds on the list:
- We find it very helpful to sort findings by severity (saving the per-task
findings for the protocol section). Attaching recommendations, as you have
done, is very helpful.
- At least in the US, everyone is very sensitive about protecting the
identity of participants. We try to take out any personally-identifiable
information - even gender - out of all parts of the report, and refer to
participants as P1, P2, and so on.
- For studies with small to medium numbers of participants, we prefer to use
ratios (3 out 10 users) in reporting, instead of percentages (30%). If
anyone still has questions about sample sizes compared to traditional
marketing approaches - well, there's an educational opportunity.

Your notes and time-on-task measurements are very detailed - what recording
methods do you use? We used to be very heavy into TechSmith Morae, but with
the time demands of agile projects find ourselves moving more and more
towards discount usability methods...

Stan Drozdetski
MITRE

Re: Usability evaluation report

[Aaron Gooch <eo...@gmail.com>]

+1 danke schon

On Fri, Dec 14, 2012 at 5:50 AM, Matthias Niederhausen <
matthias.niederhausen@t-systems-mms.com> wrote:

> Hello Stan,
>
> sorry for getting back so late, but I was ensnared in yet another user
> study
> (with some different focus) in China for the last few weeks. Hopefully, I
> will be able to post some results we got as a "side-effect" of this new
> study soon.
> I will try to address the issues you mentioned, also considering our latest
> findings:
>
> 1) In the Omelette project, we have created several means to add widgets
> directly to a workspace using recommendations. While the quality of
> recommendations is a tricky topic, users like the ability to add new
> widgets
> without leaving their workspace. One tractable option for this might be a
> search field on the top of the page offering a direct result list so I can
> click a widget to add it to the workspace. I am inspired by the twitter
> search field here, which offers icons and additional data on the result
> (e.g., type of result) immediately.
> On the other hand, virtually all of our users did not expect the page
> layout
> to be restricted to some fixed number of columns (that has to be edited
> separately). Nearly all of them tried to instinctively drag&drop widgets
> like icons on a desktop or views in Eclipse and became frustrated when they
> could not do so. So, the default layout of a page should be something that
> dynamically allocates space (maybe using a dynamic number of columns).
>
> 2) I am very glad to see that you agree on the feedback usability issues.
> One particular idea that I have had is that upon opening a workspace, the
> widget frames should be overlayed by the big widget icon until they have
> loaded and some "loading" indicator. This way, users will immediately know
> that their browser has not crashed and where which widget is.
>
> 3) Thank you for your concerns regarding the formalties of the report. This
> was the first usability report I have ever created and I will be glad to
> improve the structure of future reports with your hints. I have now heard
> of
> Techsmith Morae for the first time and it looks very promising. In our
> studies, however, we have also used "discount" approaches (Google Docs
> questionaires, measuring times by stopwatch, recording screen, audio and
> camera using the tool XSplit).
>
>
> All the best,
> Matthias
>
>
> -----Ursprüngliche Nachricht-----
> Von: Drozdetski, Stan A. [mailto:drozdetski@mitre.org]
> Gesendet: Dienstag, 13. November 2012 23:15
> An: dev@rave.apache.org
> Cc: Matthias Niederhausen
> Betreff: RE: Usability evaluation report
>
> Matthias, thank you for sharing this with the community! Very helpful to
> see
> not just the summary of findings, but also your entire protocol.
>
> I'm picking out two major issue groups in need of improvement:
>
> 1) Navigation to/from widget store + workflow for adding gadgets In our
> internal testing, we've seen quite a wide variety of mental models that
> people approach gadget-based interfaces with. Even folks with experience in
> other portal systems may not instinctively know which parts of the page are
> modifiable, and how to adjust the page contents vs page layout. We've tried
> video tutorials and first-time-use hints, but can't quite claim a silver
> bullet just yet.
>
> Another challenge is the separation between the widget store and the actual
> layout; it simply puts a barrier in the way of direct interface
> manipulation. I've been thinking through some solutions that show the user
> what's available without taking them off the page, but screen real estate
> becomes an issue. If you or anyone else have seen innovative solutions,
> please do share with the list.
>
> 2) Feedback for actions
> Yes, no feedback (or feedback that the user can't see) is a usability death
> spell. Same goes for long loading time. Thank  you for noting that, and I
> agree with your recommendations 100%.
>
> Couple of unsolicited thoughts on the report itself - in case you're
> interested in what worked/didn't work for us at MITRE, or if there are
> other
> usability nerds on the list:
> - We find it very helpful to sort findings by severity (saving the per-task
> findings for the protocol section). Attaching recommendations, as you have
> done, is very helpful.
> - At least in the US, everyone is very sensitive about protecting the
> identity of participants. We try to take out any personally-identifiable
> information - even gender - out of all parts of the report, and refer to
> participants as P1, P2, and so on.
> - For studies with small to medium numbers of participants, we prefer to
> use
> ratios (3 out 10 users) in reporting, instead of percentages (30%). If
> anyone still has questions about sample sizes compared to traditional
> marketing approaches - well, there's an educational opportunity.
>
> Your notes and time-on-task measurements are very detailed - what recording
> methods do you use? We used to be very heavy into TechSmith Morae, but with
> the time demands of agile projects find ourselves moving more and more
> towards discount usability methods...
>
> Stan Drozdetski
> MITRE
>
>