CMT4240: Designing Interactive Systems

	Lecture 2
	Identifying Users' Domain Knowledge
	Elke Duncker

Aims

	After this session you will be able to
		Appreciate the importance of domain knowledge
		Describe a range of techniques for gathering data about users' domain knowledge
		Discuss the strengths and weaknesses of each
		Be in a position to choose techniques appropriate for a particular project

Understanding context

	To improve usability need to go beyond the interface…
	Need to understand the context our designs will be used in
	As well as understanding context, need to know how the user acts in context
		Domain knowledge: knowledge about the environment and task

Domain knowledge

	Knowledge people have about context
	What to people know about?
		current state
		"facts" about the world and their job
		facts about the devices and systems they use
		ways of carrying out tasks
		personal history
		Make more clearly the distinction between domain and device

Knowledge elicitation difficulties

	Level of operation
		experts often do things rapidly and unconsciously
	Articulation
		knowledge the person doesn't know they have; common knowledge
	Overlearning
		taken for granted; sometimes better to look at beginners
	Dislocation
		between "explicit" and "tacit" knowledge

Identifying domain knowledge

	Questions to ask
		purpose of investigation?
		users of the to-be-designed system?
		constraints?
		appropriate techniques?

Example …

	Design an interactive information system to help tourists and visitors to London
	What kinds of knowledge might users possess?
	How can this be useful in design?
	How can we go about finding out?

Possible research methods

	Many different techniques for analysing interaction
		Interviews
		Questionnaires
		Observation
		Protocol analysis
		Experiments
		Log files, Workshops, Scenarios, …..
	Different methods yield different information about users' knowledge

Interviews -1

	To find out what people what know about their work - ask them!
	Can be highly flexible and responsive
	Can be more or less structured
		Some planning in advance necessary

Interviews - 2

	May be carried out “In context”
	Focus of interviews:
		critical incidents
		scenarios
	Typically involves audio recording for subsequent transcription and analysis
	Good for knowledge people are able to articulate
		less good for expert, tacit knowledge

Questionnaires

	Often used for evaluating systems, finding out about attitudes, preferences, surveying user population, …
	Fixed pre-planned set of questions
		Not flexible or responsive
	What kinds of information are elicited using questionnaires?

Questionnaires - 2

	Benefits of  questionnaires?
		Cheap
		Allows comparisons and analysis
		Cross check and validate other data (interviews)
	When are they more useful than interviews?
	Generally poor way of eliciting users' knowledge

Observation

	People often not good at explaining what they know, so
	Watch what people actually do in their natural environment
		Often as a participant
	Build up a rich understanding of knowledge used, social organisation, context, …

Observation - 2

	Tends to be in-depth, and looks at only a small number of users
	Good or identifying what people do
	Identifying why and what knowledge is used typically time consuming:
		Ethnographic studies may take months, involve "immersion" in field
		"Discount" techniques require less time. E.G., Contextual inquiry

Protocol Analysis

	Allows analyst to identify when and how in an interaction user’s knowledge is used
	Involves observing user activity, making notes or video / audio record
		Common forms:
			“Think aloud protocol”, “retrospective protocol”
		Analyst may prompt user
		Recordings are transcribed and coded using a specialised coding scheme
	Not applicable in all situations

Protocol Analysis

	Questions and prompts:
		How do you….?
		What are you trying to do?
		What will happen if….?
		What has the computer done now?
		What does this message mean?What did you expect to happen?

Controlled Experiments

	Involves formulating hypothesis
	Carried out under carefully controlled conditions
	Useful for comparing design alternatives (for example) and usability studies
	Not useful for exploring users’ domain knowledge

Comparison

	Interviews
		Rich source of data; Relatively quick to administer; Flexible; Focus on what people can talk about
	Questionnaires
		Data may be easy to process; Quick to administer; Rigid; Allow large samples and comparison; Often poor for knowledge elicitation

Comparison - 2

	Observation
		Rich data; Time consuming to elicit users’ knowledge; Focus on what people do; Analysis may be complex
	Protocol analysis
		Analysis may be complex; Requires some skill of researcher; Requires users to talk and act; Useful for eliciting users' knowledge

Mixing Techniques

	Techniques have different strengths & weaknesses and can explore different aspects of domain knowledge
	Using several techniques can help to build up a richer picture
	“Triangulation”
		Verify findings using several techniques and different samples

Case Study: Emergency Ambulance Dispatch

	Project to understand decision making in ambulance dispatch and critique proposals for new technology
	Large and complex system
		many people with different jobs
		computer systems
		outside world
	Used a combination of techniques to analyse from different angles and validate findings

Slide 22

Techniques Used in the Study

	Initial interviews
		Open, informal interviews with senior management for background
	Observation
		Notes and video record of several sessions
		Later transcription
	Interviews
		Critical incident analysis
		Contextual interviews

Results

	Controllers possess rich knowledge
		About geography
		About state of the world
		About previous incidents & clinical decision making
	Knowledge about computer systems also required
	Fit between technology and domain tasks?

Summary

	Context and domain knowledge is important
	Can inform design
	Many techniques exist to analyse knowledge users possess
	Can be used together to provide richer data and cross check

Further Reading

	Dix
	Preece
	Shneiderman