#idm21

Can be used at both ends of the process:

Beginning: Requirements gathering activity

- Observation of existing system in use

- Trial of putative design solutions

During: Formative evaluation

- Lo-fi and alpha prototypes

End: Summative evaluation

- Observation of beta prototype and production version in use

Gathering empirical data from usage of the software by representative users

Paper prototype stage

Software prototype stage (Alpha) (Functionality may be ‘Wizard of Oz’)

Initial release stage (Beta)

Long term evaluation of installed system

Process and requirements:

1.  Subjects are set tasks to accomplish using the system under test

Based on scenarios collected during requirements analysis phase

Background training, motivation and data set up

A precise script should not be given

Task should be encountered as it would in real use

2.  Observed attributes of the performance are recorded

Time to complete

Mistakes and errors made

Methods used

Expressions of confusion or frustration

Comments on acceptability, satisfaction and conformance to users mental-model

3. Data from sessions analysed

Data prepared:

Notes and commentary transcribed

Recorded video tagged

Data reviewed to extract information:

Occurrence of critical incidents (this is normally when a user is brought to a halt)

Timings and counts of events

Expressions of dissatisfaction (the number of times an action is completed)

NB Make sure the people who have designed the interface are not visible or become involved with the people testing.

4. Result presented

Formative evaluation

Developers or designers directly observe sessions

Notes and summaries

Video presentation of highlights / low-lights

Summative evaluation

Formal report might well be required for sign-off and checking back on what the initial brief / requirements were

BS ISO/IEC 25062:2006 Software engineering – Software product Quality Requirements and Evaluation (SQuaRE) – Common Industry Format (CIF) for usability test reports

How to run a session:

1. Running a session – Preparation

Design sessions

Establish test objectives

Health & Safety Risk Assessment

Design and write protocol

Prepare consent forms

Prepare test material

Book facilities

Book staff

Facilitator

Recording technician

Receptionist

Recruit subjects

Establish type and number required

2. Running a session - Managing the session

Subject greeting and introduction

Obtain nondisclosure agreement signature (where required)

Explain purpose and procedure - N.B. The software is under test, not the subject

Establish that the participant can withdraw at any time

Obtain consent form signature

Give reward (if any – and get receipts)

Obtain necessary demographic information

Facilitate task/s

Execute protocol in a consistent manner

Be supportive but not directing

Allow subject to give up

Encourage commentary (where protocol requires)

Manage time

Conclude session

Obtain reflection/comments (where protocol requires)

May involve subject reviewing recording of performance

Request that subject does not discuss specifics of test with colleagues (where necessary)

Thank subject for their contribution

3.  Running a session – Securing and managing data

Secure data immediately after a session

Complete your own notes

Collect notes from observers

Ensure notes are complete and readable

Ensure any recording was successful

Label and back-up recordings

Approaches

– Straight observation

– Subject gives ongoing commentary

– Subject gives a post-task commentary

– Subject questioned during task

– Structured interview/questionnaire on completion of task

– Subject reviews recording of session and gives commentary

– Pair discovery eg a couple booking something together

Legal/Ethical issues

No undue stress both physical and mental

The purpose of the testing must be adequately explained and assurances of confidentiality must be authentic

Workstations must be set up in compliance with appropriate standards and legislation

Length of sessions and intensity of activity must be acceptable

Subjects must be able to request a break or early termination of the session

Refreshments should be available

– Children as subjects

Be aware of legislative requirements

Agreement given by parent or guardian

Recruitment usually through organisation (school, club, etc.)

– People with disabilities

Mental capability – informed consent must be obtained

Motor or sensory disability – particular attention must be paid to health and safety concerns

Recruitment must be managed appropriately

– Professionalism (duty of care to employer)

Impartiality and objectivity of tester

Economic efficiency of operation

Usability inspection method

The cognitive walkthrough method is a usability inspection method used to identify usability issues in interactive systems, focusing on how easy it is for new users to accomplish tasks with the system. Cognitive walkthrough is task-specific, whereas heuristic evaluation takes a holistic view to catch problems not caught by this and other usability inspection methods. The method is rooted in the notion that users typically prefer to learn a system by using it to accomplish tasks, rather than, for example, studying a manual. The method is prized for its ability to generate results quickly with low cost, especially when compared to usability testing, as well as the ability to apply the method early in the design phases, before coding even begins.

Assumptions:

User has a rough plan

User explores the system, looking for action to contribute to performance of action

User section action that seems best for the goal

User interprets response and asses whether progress has been made toward completing a task

Evaluators are provided with: (3/5 people as 1 is not enough)

description of the prototype of the system. Not necessarily complete but detailed (this can be done quite early on)

description of a representative task a user is to perform

detailed list of the option actions needed to completed the taskin with the given prototype

an indication of who the users are and what kind of experience and knowledge the evaluators can assume about them

Walking through the tasks

For each action, the evalulators answer the following four questions:

Will the user try to achieve the effect that the subtask has? Does the user understand that this subtask is needed to reach the user's goal?

Will the user notice that the correct action is available? E.g. is the button visible?

Will the user understand that the wanted subtask can be achieved by the action? E.g. the right button is visible but the user does not understand the text and will therefore not click on it.

Does the user get appropriate feedback? Will the user know that they have done the right thing after performing the action?

By answering the questions for each sub task usability problems will be noticed.

Common mistakes In teaching people to use the walkthrough method, Lewis & Rieman have found that there are two common misunderstandings:

The evaluator doesn't know how to perform the task themself, so they stumble through the interface trying to discover the correct sequence of actions—and then they evaluate the stumbling process. (The user should identify and perform the optimal action sequence.)

Summary: Jakob Nielsen's 10 general principles for interaction design. They are called "heuristics" because they are broad rules of thumb and not specific usability guidelines. You could give someone to work through a wireframe or low-fi prototype and work through it. You can either sit with someone and ask them to talk through what they are doing / seeing. Session can last 1/2 hours max and the evaluator should never argue the users points, only explore them further.

During the session:

Evaluators goes through the interface several times

First pass to get a feel for the flow and scope of the system

Second pass to focus on specific elemtns

For non-domain expert evaluations, supply a task scevario

Inspect dialogue with heuristic principles / or others

Comments should be specific and rationalised, 'I don't like the colour' is not useful feedback - always ask why!?!

After evaluation, aggregate the results, if there are many people saying the same feedback, it is probably a problems - access the scale

Consider holding a debriefing session in brainstorming mode - evaluators, observers and resps of the design time to look at possible solutions

Ensure their is positive feedback as well

Visibility of system status The system should always keep users informed about what is going on, through appropriate feedback within reasonable time. (You can see this when users look elsewhere for information, eg to see if a package / page is hanging - users need to see things like progress bars.)

Match between system and the real world The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

User control and freedom Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo. (Apple have always done this well, it will allow users to make mistakes and recover from the gracefully. This allows user to feel more confident if they know they can go back. If you know you are likely to 'fail' you are less likely to explore.)

Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions. (Ensure consistency internally and externally, make sure the word and layout are the same within the site / software itself or that yours will fit in with other application / site. This could be extended to using GUI guidelines for brand.)

Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action. (All syntactic* errors should be detectable at this stage, so you can't put incorrect data into fields. You can always ask users to confirm a possible error - think MailChimp's 'type DELETE' in order to fully delete something.)

Recognition rather than recall Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. (This is used by Police for Identikit pictures, it is very difficult to be very specific verbally, however if you're presented with a set of options it is easier to choose. This phenomenal power of recognition is something we should take advantage of but also be wary of. Should information be carried through, eg addresses.)

Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions. (This is a moving target as each 'use' is not the same, as the user will lean from their first use. Examples of this include: customisable user interfaces; short-cut keys; function keys - will it always be the same for advanced versus new users?.)

Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility. (Screen junk can distract from use, what is a button versus a flourish. Consider if the design of the website is conflicting with the content - consider Google.)

Help users recognize, diagnose, and recover from error Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. (Help users recognise, diagnose and recover from errors, make certain that the interface updates in an appropriate manor. If the systems sees you've made an error.)

Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

*Syntactic and Semantic Errors

Syntax errors occur during the parsing of input code, and are caused by grammatically incorrect statements. Typical errors might be an illegal character in the input, a missing operator, two operators in a row, two statements on the same line with no intervening semicolon, unbalanced parentheses, a misplaced reserved word, etc.

Nielsen's Heuristics Evaluation

A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the "heuristics").

This came out of Nielsen's PHD research, of trying to find a cost-effective way of doing user testing. There are probably (hundred) of thousands of useability experts in the world versus billions of web pages, apps and games therefore the economics don't stack up.

However, not all experts are equal.

Illustration showing which evaluators found which usability problems in a heuristic evaluation of a banking system. Each row represents one of the 19 evaluators and each column represents one of the 16 usability problems. Each square shows whether the evaluator represented by the row found the usability problem represented by the column: The square is black if this is the case and white if the evaluator did not find the problem. The rows have been sorted in such a way that the most successful evaluators are at the bottom and the least successful are at the top. The columns have been sorted in such a way that the usability problems that are the easiest to find are to the right and the usability problems that are the most difficult to find are to the left.So what number of evaluators do you need?

Curve showing the proportion of usability problems in an interface found by heuristic evaluation using various numbers of evaluators. The curve represents the average of six case studies of heuristic evaluation. Curve of number of usability problems found as more evaluators are added Nielsen and Landauer (1993) present such a model based on the following prediction formula for the number of usability problems found in a heuristic evaluation.

The sweet spot for the number is around 3/4 - you will never get 100% of the problems resolved but this will place you as close as possible.

Information 'energy'

Kinetic v potential - involves momentum eg from high to low (entorpy)

Dynamic v static

Algorithm v data structures - eg for unstructured data you need a more complex algorithm and vice versa

Task analysis v content analysis - the activity and how this is held on the site, what is the value to the user

Form v content - presentation of media eg form of text; video; images

Interaction design v information architecture - the information architect increased the potential for the interaction design to make it work or use it

"Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones."

Donald Rumsfeld

Information seeking strategies:

Think about this in the context of asking questions.

1. Known-items seeking (known unknowns)

- you know what you are looking for, how to articulate requirement, where to find it. eg recipe for cooking lasagna

2. Explanatory seeking (unknown unknowns)

- not exactly sure what you're looking for, doesn't know how to articulate thee requirement (involves learning about the domain). But, you will know when you see it eg what shall I have for dinner?

3. Exhaustive research (unknown knowns)

- requires all the information on a particular topic, in this instance it is impossible to determine all of the variations (without a very complex algorithm) eg list all of the ways to make a lasagna (insurance websites are a good example of this)

4. Re-finding (known knowns)

- relocating previously found information, this could be search, history, cmd f (on a mac) search, emails etc...

Consider George Kingsley Zipf Zipf was an American linguist and philologist who studied statistical occurrences in different languages.[2

…an information [seeker] will tend to use the most convenient search method, inthe least exacting mode available.Information seeking behaviour stops as soon as minimally acceptable results are found. Wikipedia, Principle of least effort

If you present an easier option, such as a search on a website, users will use it. This type of approach is a decision-making strategy or cognitive heuristic that entails searching through the available alternatives until an acceptability threshold is met; this is called satisficing. And the opposite behavior to extensively adding more, and more detail.

Definition form the Information Architecture Institute:

The structural design of shared information environments.

The art and science of organising and labeling websites, intranets, online communities and software to support usability and findability.

An emerging community of practice focused on bringing principles of design and architecture to the digital landscape.

a sign is something, A, which brings something, B, its interpretant sign determined or created by it, into the same sort of correspondence with something, C, its object, as that in which itself stands to C.

Peirce

Sign: something that represent something to a perceive (ie someone who experiences the interpretant)

He breaks down these elements to things we see already within our environments into a type of: icon; index; symbol

Icon - something that resembles the object to the perceiver, for example a photograph

Index - there is an actual connection between the sign and the object, for example if we press our laptop button on there is an index for the action to happen

What is information?

Facts provides about something or someone - a vital piece of information

What is conveyed or present by a particular arrangement or sequence of things

Computing data as processed, stored, or transmitted by a computer

(Information) "A difference which makes a difference"  Widely attributed to Gregory Bateson (1904-1980)

The content is not just the sequence of words, it is the impact it has upon a person.

When we're designing what is the difference we're going to make when designing for our recipient.

Tufte's information design principles:

1. Principles of graphical excellence: substance, statistic and good design forms the basis.

It should give the viewer the greatest number of ideas in the shortest time with the lease ink in the smallest space.

Requires telling the truth about data

2. Principles of graphical integrity

The representation of numbers, as physically measure on the sure face of the graphic itself, should be directly proportional to the  numerical quantities represented

Lie factor = (size of effect shown in the graphic) / (size of effect in data)

Clear, detailed and thorough labeling should be used

Show data variation not design variation!

3. Data-ink

- this is the non-erasable core of a graphic, the non-redundant ink arranges in response to the variation in the numbers represented

- is what you're doing just decorative

- erase non-data ink

- erase redundant data-ink

4. Chartjunk

- Refers to over-busy grid lines and excess tricks and debris

- check if moire vibration occurs

- ensure the graphic looks as it should and what it is supposed to be

5. Data density

- don't make items bigger than they need to be

6. Small multiples

- well design small multiples are: comparative, multivariate; shrunken

7. Aesthetics & Techniques

- what typre of chat pie, bar etc.

8. Friendly data graphics

- runs from left to right,

- labels on graphic not legend so people can read it

- clear type and colours

- graphics 50% taller than wide

- look for redundancy, what is repetitive