User Research Methods and Recommendations

Advisory
This document is intended as a high-level overview of the concepts, methods, benefits, and challenges of user research as a discipline within an Information Technology organization.
 
user_research_advisory.pdf287 KB
Authors
  • Timothy Murray (Enterprise Architecture)
Version 1.0
Last Revised 3-Feb-2022
Status Released
Document Type Single Topic Guidance
Audience Level
  • IT Director / Manager
  • Solution Architect and Project Manager
  • Application Developer and Designer
  • DevOps Staff
  • Senior IT Engineers
  1. Purpose of this Document

    This document is intended as a high-level overview of the concepts, methods, benefits, and challenges of user research as a discipline within an Information Technology organization. This advisory encourages individual contributors involved in the discovery, design, development, and implementation of new products, processes, or services; or the monitoring of success criteria for existing solutions, to incorporate appropriate research methods as part of their regular strategic and operational practices. Links to additional resources, including guides for performing select user research methods, are provided.

  2. Executive Summary

    Understanding the human dimensions that shape a person’s interactions with an IT organization—and the solutions it delivers—are critical for the successful delivery of value to the people it ultimately serves. Integrating user research methods with routine IT procedural activities (e.g. requirements gathering, solution design, user acceptance testing) enables the organization to:

    • unveil unmet needs and improve strategic decision making
    • measure the value and impact of current or proposed solutions

    Potential user research methods that supplement existing IT activities include user interviews, surveys, expert reviews, usability tests, and the collection of experience metrics through usage analytics.

    The proposed user research methods provide opportunities for participation from everyone in the IT organization, ensuring that the organization and its individual contributors:

    • never lose sight of who they are serving
    • appreciate and plan for the human impact of strategic and tactical decisions throughout the IT lifecycle
  3. Recommended User Research Methods for an IT Organization

    There are many opportunities for user research to augment and improve the successful delivery of solutions. While many User Experience methods and techniques are focused on gathering insights of the human impact of technology decisions, a few recommended methods should be used by technologists to develop a richer understanding of their end-users.

    The inclusion of one or more of the following research methods into routine IT practices will assist with defining and developing new solutions, as well as the evaluation and improvement of existing technologies and services.

    1. User Interviews

      Traditional business activity to supplement: Discovery and requirements gathering activities

      Interviews with representative end-users, as well as business stakeholders, are an invaluable source for gathering information about people’s needs and goals; their motivations, thoughts, and decision-making processes; and their attitudes and beliefs. The User Interview method lends itself to allowing broad participation from technology teams and business stakeholders, as a way of information gathering and empathy building (i.e. hearing and observing first-hand from people is a natural and effective way to make connections with the people who utilize your products and services).

      It is also important to note that interviewing business stakeholders as part of a user interview process is highly recommended. Knowing and understanding the business goals more deeply, as well as its motivations and expectations, will help find gaps and overlaps between business and user needs, provide a path to stakeholder alignment, and assist in the process of better defining problems and imagining solutions.

      IMPORTANT: User Interviews are not the same as requirements gathering. Although requirements gathering is a traditional way for technologists to assess user needs, the process tends to focus more on the capabilities and attributes of a proposed solution, as opposed to exploring and evaluating the human dimensions of cognition, emotion, and behavior; which can lead to greater insights for producing innovative and alternative solutions.

      Sample research questions answered by this method

      • What are the needs and goals of the users and the business?
      • What are the processes currently used to solve problems?
      • What are the expectations of users and the business?
      • Who is the right population for a solution, and are there additional audiences who need to be considered?
      • How do business goals align with user needs and expectations?
    2. Surveys

      Traditional business activities to supplement: Discovery and requirements gathering activities

      Surveys are another way of gathering information about users that can reveal similar human dimensions as User Interviews for research focused on new and existing solutions, with the added benefit of scalability, which can help to surface the size or importance of a particular aspect. However, due to their one-way nature, surveys limit the depth to which the researcher can probe on areas of interest, and don’t elicit the level of empathy that a two-way conversation can produce.

      Sample research questions answered by this method

      • How big is the problem or opportunity?
      • How satisfied are people with an existing solution?
      • What is the desire for a new product or service?
    3. Expert Reviews

      Traditional business processes to supplement: Design and development, Implementation, Quality Assurance (QA), User Acceptance Testing (UAT)

      An effective method for identifying baseline usability issues is to have an IT professional trained in this method or a User Experience practitioner perform an Expert Review, sometimes referred to as a Heuristic Review—a systematic review of a solution, which looks to identify any violations of common UX principles and best practices. This method can surface critical issues, but its effectiveness is limited by the reviewer’s knowledge of usability best-practices, and the context and subject-matter involved with the solution.

      Sample research questions answered by this method

      • Are there issues with a solution that will likely create barriers for end-users?
    4. Usability Tests

      Traditional business process to supplement: Design and development, Implementation, Quality Assurance (QA), User Acceptance Testing (UAT)

      Testing concepts, prototypes, and developed solutions can provide information about how a solution performs, and whether it is an adequate fit-to-need for intended users. These tests can have a quantitative dimension—measuring what users do, such as how long it takes to complete a task, and what kinds of user errors are repeated (e.g. navigation patterns that cause a user to become ‘lost)’—as well as qualitative dimensions, which help the technologist to better understand user comprehension and why users behave in certain ways.

      IMPORTANT: Usability testing is not Quality Assurance (QA) or User Acceptance Testing (UAT). QA and UAT can surface issues around whether a proposed solution meets necessary technical aspects, and answer questions such as ‘Did we build the thing right?’, as opposed to the more holistic, user-centered approach, which can get the technologist closer to understanding “Did we build the right thing?”

      Sample research questions answered by this method

      • Does the solution deliver the value to the user as intended?
      • Are there elements of the solution that prevent the user from satisfying their needs?
      • What opportunities for improving user utility, efficiency, and satisfaction exist?
    5. Usage Analytics

      Traditional business process to supplement: Monitoring of success criteria and KPIs

      Measuring the utilization of a current service or product with the additional of experience metrics can reveal important insights into how effective and efficient the solution is at serving both user and business goals.

      Sample research questions answered by this method

      • How many people are using a current solution?
      • How are people using a current solution?
  4. Discussion

    1. Why Conduct User Research?

      For IT solutions to be utilized and return value to the organization, it is critical that the human dimensions of end-users—cognition, behavior, and context—be placed at the center of design, development, and decision-making processes.

      The Human Dimensions of End-Users:

      • User Cognition

        What people think and know, including their needs, goals, values and beliefs, expectations, motivations, feelings, and perceptions.
      • User Behavior

        What people do, including their actions and habits.
      • User Context

        The setting which a person uses a product or service, including their environment and constraints.

      Uncovering these human insights helps improve both strategic and tactical decisions by rooting them in empathy and understanding for the people who will ultimately use and benefit from a product or service; helping to mitigate the pitfalls of assumption and bias, by relying on observation rather than anecdote and preconceived beliefs.

      By refocusing success from the successful deployment of solutions to a more user-centered, empathetic lens focused on the human dimensions of a solution, the IT organization can better plan for what and how it delivers value to its customers. User Research is the necessary, evidence-based means for uncovering the human insights that make empathetic, user-centered decision-making possible.

    2. Benefits and Challenges of User Research

      Benefits of User Research

      • Moves an IT organization to being more planful and strategic by surfacing currently unmet needs and goals
      • Increases adoption of systems and processes by making sure the solution fits the needs, behaviors, and beliefs of the end-users
      • Accelerates consensus building and decision-making, by using evidence as opposed to anecdote
      • Validates or invalidates assumptions and hypotheses before they become ingrained in an initiative’s DNA
      • Continuously checks solutions as new information is gathered, and keeps the solution space open to alternatives
      • Creates efficiency, by allowing a solution to pivot earlier in the process, saving time and resources

      Challenges of User Research

      • Research can be time consuming and resource intensive, depending on the method and breadth of the inquiry
      • Research is not an integrated part of current planning or development methodologies in many IT organizations
      • Some (not all) methods require levels of expertise and capabilities that may not be developed yet within the organization
    3. Types of User Research

      User Research can be roughly divided into two lines of inquiry: one that looks to design or find new solutions and opportunities, and one that looks to evaluate existing solutions.

      • Generative Research

        The examination of people’s behaviors, thoughts, feelings, and attitudes to develop a human-centered solution space for defining, developing, or procuring new products and services.
      • Evaluative Research

        The study of how people interact with existing technologies and processes to measure their utility, understand their impact, and determine their fit with users’ needs and goals.

      Within both of these general branches of research are further categories of research types that can help us to understand different dimensions of the human experience:

      • Attitudinal Research

        Gives us a deeper understanding of how people think and feel
      • Behavioral Research

        Provides a richer view into people’s actions and behaviors
      • Qualitative Research

        Helps us understand the “Why?” and the “How?” of a person’s experience
      • Quantitative Research

        Answers the questions of “What?” and “How many?”
    4. Common Methods of User Research

      There are many User Research methods, each of which has strengths depending on the research goals of the inquiry.

      Some common methods and their fit for research objectives are listed below.

        Generative Evaluative Attitudinal Behavioral Qualitative Quantitative
      Usability Testing X X X X X X
      Focus Groups X X X   X  
      User Interviews X X X   X  
      Contextual Inquiry X X X X X  
      Card Sorting X X X   X X
      Surveys X X X   X  
      Click-stream Analysis   X   X   X
      Expert Review   X        
      Usage Analytics   X   X   X
      1. Definitions of Methods

        • Usability testing – Allows teams to observe how individuals interact with solutions as the user steps through a series of tasks
        • Focus groups – Reveals the feelings, opinions, and attitudes of individuals, through a moderated discussion within a group context
        • User interviews – Collects first-hand evidence of user attitudes, goals, and perceptions, with the ability to re-focus the discussion and probe on areas of interest
        • Contextual inquiry – A combination of user interviews and usability testing with the added dimension of the method being performed in the user’s natural context, such as a workspace
        • Card sorting – A method of finding relationships and hierarchy of information through a process of organizing text artifacts
        • Surveys – Collects self-reported data about opinions and attitudes, thoughts, feelings, and perceptions
        • Expert review – An evaluation using agreed-upon usability standards and best-practices
        • Usage analytics – The collection and measurement of usage and experience metrics to optimize a solution
  5. Incorporating User Research into Information Technology Activities

    1. Who should be involved in User Research?

      Every role in an Information Technology organization provides inputs that impact the overall experience of products and services, including the process by which a user realizes the value of a solution, its fit-to-need, its completeness and accuracy, and its utilization. Therefore, everyone would benefit from a deeper understanding of the effects their contributions play.

      The level of effort and expertise involved in research methods varies, but a broad set of opportunities for participation by anyone in an IT organization include:

      • Facilitating Activities

        Leading a research method and making sure key research questions are addressed
      • Note Taking/Observing

        Being an eyewitness to a method, either in real-time, or through recordings and/or transcripts
      • Research Synthesis

        Taking part in the sense-making of the evidence collected to generate insights
      • Research Question Generation

        Formally or informally documenting questions that arise during normal business operations, such as ‘How would someone use this?’, ‘How would someone feel about this?’

      Including everyone in the User Research process helps to ensure that the organization doesn’t lose sight of who it serves and why it matters.

    2. When should User Research be performed?

      Ideally, User Research methods would be employed at two key points in an IT organization:

      • When measuring performance of an existing solution
      • When evaluating and designing changes to existing or new solutions

      In addition, User Research should be performed whenever there are questions or assumptions about user needs or behaviors driving IT decision making, or if the human impact of decisions is not well understood.

      The need for User Research exists along the entire IT lifecycle, from strategic planning to the design, development, and procurement of new solutions, to the operational maintenance and ultimate decommissioning of existing products and services.

      At a minimum, User Research activities should be incorporated in an integrated way into planned activities surrounding:

      • the discovery and analysis of new solutions and processes
      • the design, development, procurement of new solutions
      • the evaluation of existing solutions
    3. How much User Research is enough?

      While User Research should be an ongoing process embedded in every stage of the IT lifecycle, most individual methods applied to specific research goals will eventually reach a point at which new information is no longer providing meaningful insights for the research objective(s). These diminishing returns can generally be mitigated at the practical level by:

      • producing enough evidence that patterns begin to appear
      • stopping when no new patterns are being introduced (i.e. the results are saying the same thing over and over).

      However, the answer to how much User Research should be conducted for any initiative at the strategic level should take into consideration:

      • the risks of making IT decisions with too little information about the people who will be affected
      • the costs of performing and making sense of collected information
      • the potential impacts and benefits to both the user and the organization.
  6. Conclusion

    By layering an understanding of the human dimensions of cognition, behavior, and context to IT activities, User Research helps IT organizations create better outcomes for the people it serves. User Research supports the organization in its strategy and decision-making by revealing new opportunities, improving decision making, and reducing uncertainty—through the collection of evidence, and testing and validation of assumptions and hypotheses—allowing it to make decisions more confidently and efficiently regarding the value and the impact of its activities.

  7. User Research Guides and Resources

    The following resources are intended as a starting point for learning and putting into practice the UX methods described in this advisory.

    1. Putting Methods into Practice

      1. User Interviews

        Hall, Erika. “Interviewing Humans.” A List Apart, 10 Sep. 2013, http://alistapart.com/article/interviewing-humans/
      2. Surveys

        Jarrett, Caroline. “Surveys That Work: The Book and Extras.” Effortmark, 13 Oct. 2021, www.effortmark.co.uk/surveysthatwork.
      3. Expert Reviews

        Nielsen, Jakob. “10 Usability Heuristics for User Interface Design.” Nielsen Norman Group, 24 Apr. 1994, www.nngroup.com/articles/ten-usability-heuristics.
      4. Usability Testing

        Usability Testing - UX @ Harvard Library - Harvard Wiki, https://wiki.harvard.edu/confluence/display/UHL/Usability+Testing.
      5. Usage Analytics

        Walker, Alice Emma. “The Ultimate Guide to Google Analytics for UX Designers.” UX Collective, 7 July 2018, https://uxdesign.cc/google-analytics-ux-alice-emma-walker-958d6f0f0af3.

        “Usage Analytics in User Research.” Think Design, 9 Jan. 2020, https://think.design/user-design-research/usage-analytics.
    2. UX Resources at Harvard

      1. Harvard Web Publishing

        Harvard Web Publishing offers a full suite of UX services including research, design, and testing: https://hwp.harvard.edu/user-experience-ux-and-design.
      2. UX @ Harvard Library - Harvard Wiki

        The UX @ Harvard Library - Harvard Wiki has a curated collection of User Research method templates, case studies, and other helpful tools: https://wiki.harvard.edu/confluence/display/UHL/UX+@+Harvard+Library.
      3. User Research Center at Harvard

        In addition to a robust lab for usability testing, The User Research Center also offers advice and consultation on recruiting participants, conducting UX methods, and using UX and Accessibility tools and technologies: https://urc.library.harvard.edu/
      4. Digital Accessibility Services

        Digital Accessibility Services (DAS) offers training, consultation, and resources to support accessible web content creation and development: https://accessibility.huit.harvard.edu/resources

a54d3b3f70a15a25e8eef64048e10d7f