Notes

Outline

CMT4240: Designing Interactive Systems Lecture 2 Identifying Users' Domain Knowledge Elke Duncker

Aims After this session you will be able to Appreciate the importance of domain knowledge Describe a range of techniques for gathering data about users' domain knowledge Discuss the strengths and weaknesses of each Be in a position to choose techniques appropriate for a particular project

Understanding context To improve usability need to go beyond the interface… Need to understand the context our designs will be used in As well as understanding context, need to know how the user acts in context Domain knowledge: knowledge about the environment and task

Domain knowledge Knowledge people have about context What to people know about? current state "facts" about the world and their job facts about the devices and systems they use ways of carrying out tasks personal history Make more clearly the distinction between domain and device

Knowledge elicitation difficulties Level of operation experts often do things rapidly and unconsciously Articulation knowledge the person doesn't know they have; common knowledge Overlearning taken for granted; sometimes better to look at beginners Dislocation between "explicit" and "tacit" knowledge

Identifying domain knowledge Questions to ask purpose of investigation? users of the to-be-designed system? constraints? appropriate techniques?

Example … Design an interactive information system to help tourists and visitors to London What kinds of knowledge might users possess? How can this be useful in design? How can we go about finding out?

Possible research methods Many different techniques for analysing interaction Interviews Questionnaires Observation Protocol analysis Experiments Log files, Workshops, Scenarios, ….. Different methods yield different information about users' knowledge

Interviews -1 To find out what people what know about their work - ask them! Can be highly flexible and responsive Can be more or less structured Some planning in advance necessary

Interviews - 2 May be carried out “In context” Focus of interviews: critical incidents scenarios Typically involves audio recording for subsequent transcription and analysis Good for knowledge people are able to articulate less good for expert, tacit knowledge

Questionnaires Often used for evaluating systems, finding out about attitudes, preferences, surveying user population, … Fixed pre-planned set of questions Not flexible or responsive What kinds of information are elicited using questionnaires?

Questionnaires - 2 Benefits of  questionnaires? Cheap Allows comparisons and analysis Cross check and validate other data (interviews) When are they more useful than interviews? Generally poor way of eliciting users' knowledge

Observation People often not good at explaining what they know, so Watch what people actually do in their natural environment Often as a participant Build up a rich understanding of knowledge used, social organisation, context, …

Observation - 2 Tends to be in-depth, and looks at only a small number of users Good or identifying what people do Identifying why and what knowledge is used typically time consuming: Ethnographic studies may take months, involve "immersion" in field "Discount" techniques require less time. E.G., Contextual inquiry

Protocol Analysis Allows analyst to identify when and how in an interaction user’s knowledge is used Involves observing user activity, making notes or video / audio record Common forms: “Think aloud protocol”, “retrospective protocol” Analyst may prompt user Recordings are transcribed and coded using a specialised coding scheme Not applicable in all situations

Protocol Analysis Questions and prompts: How do you….? What are you trying to do? What will happen if….? What has the computer done now? What does this message mean?What did you expect to happen?

Controlled Experiments Involves formulating hypothesis Carried out under carefully controlled conditions Useful for comparing design alternatives (for example) and usability studies Not useful for exploring users’ domain knowledge

Comparison Interviews Rich source of data; Relatively quick to administer; Flexible; Focus on what people can talk about Questionnaires Data may be easy to process; Quick to administer; Rigid; Allow large samples and comparison; Often poor for knowledge elicitation

Comparison - 2 Observation Rich data; Time consuming to elicit users’ knowledge; Focus on what people do; Analysis may be complex Protocol analysis Analysis may be complex; Requires some skill of researcher; Requires users to talk and act; Useful for eliciting users' knowledge

Mixing Techniques Techniques have different strengths & weaknesses and can explore different aspects of domain knowledge Using several techniques can help to build up a richer picture “Triangulation” Verify findings using several techniques and different samples

Case Study: Emergency Ambulance Dispatch Project to understand decision making in ambulance dispatch and critique proposals for new technology Large and complex system many people with different jobs computer systems outside world Used a combination of techniques to analyse from different angles and validate findings

Slide 22

Techniques Used in the Study Initial interviews Open, informal interviews with senior management for background Observation Notes and video record of several sessions Later transcription Interviews Critical incident analysis Contextual interviews

Results Controllers possess rich knowledge About geography About state of the world About previous incidents & clinical decision making Knowledge about computer systems also required Fit between technology and domain tasks?

Summary Context and domain knowledge is important Can inform design Many techniques exist to analyse knowledge users possess Can be used together to provide richer data and cross check

Further Reading Dix Preece Shneiderman