Evangelizing the business value of user research (especially in enterprise software teams)

Executive summary

The dedicated User Research team within a healthcare company presented to executives, product managers, and developers not about the processes that user research encompassed, but about the benefits from engaging with and conducting user research: confidence in opportunity-cost tradeoffs, data to face opinions with, ready delegation, never needing to duplicate effort, and more.

The positioning was well received, and every software development team in the organization eventually grew to incorporate or improve their research capabilities, advancing their position in the design research maturity model.

The ~40m talk is now publicly available.

Design research maturity model

In May 2016, Chris Avore of NASDAQ Design published their “design research maturity model,” intended to serve as “a helpful signpost indicating where you are on your journey to working where design research is primary tool to discovering new insights, validating existing hypotheses, and innovating into new and existing markets.”

(download original in Keynote or Powerpoint)

At Advisory Board, we found we could trace every one of our own product teams’ research maturity over the past three years in this model, and recognized our own internal efforts had paralleled Chris’ at NASDAQ.

Prior to his joining NASDAQ in 2011, there were no product or user experience designers in the organization. Growth in their product portfolio was through acquisition, and design and research were ad hoc efforts conducted by product management and developers.

Five years on, NASDAQ Design is recognized both within:

Wherein the CEO specifically cites product design as a differentiator on a *quarterly earnings call*.

— Chris Avore (@erova) April 27, 2016

(NASDAQ Q1 2016 earnings call) and outside the organization.

Their public efforts, like their Pro/Design conference and classes (see also the class wrap-up):

and the release of their internal Mosaiq research asset management software:

are signals that even traditionally non-design-centric organizations can change, and that executive management can understand and enable design to make a difference in the bottom line.

Like NASDAQ, Advisory Board did not start out with design-centric software practices. Originally a financial industry syndicated research firm, it moved into healthcare, and expanded into software and consulting. As recently as 2012, nearly all its best practices research was done for customers, not for itself on customers.

Like NASDAQ, many of Advisory’s product teams were acquisitions, with independent engineering and design practices, and few were established enough to have user experience staff at the time of their acquisition. There was a conscientious effort to not kill the software teams’ cultures by imposing Advisory’s, which had the side effect that coordination between teams was often informal.

Even in the slow-moving healthcare industry, design had become a competitive differentiator to such a degree that Advisory’s product teams had begun staffing up user experience designers and researchers on their own. In late 2014, a dedicated, centralized User Research team was formed and charged with evangelizing more rigorous, consistent, research and design practices across all our product teams.

Influence at scale

Our then-director, Erin Howard, describes the original expectations:

When we first formed the UX Research group at The Advisory Board, the intent was to make immediate, broad impacts across the suite of products we were serving to up to healthcare organizations. It was created out of need, out of the motivation to do what is right by our users, our customers.

The formation of the User Research (UXR) group pushed the company into the “Progressing” column of the maturity model, but individual product teams were still in varying stages. From some of them, we heard the same complaints found in any enterprise software shop. Erin Howard again:

Healthcare systems have been making tremendous strides even within the last five years transforming their business models and stitching together (no pun intended) software and consumerism to bring about incredibly progressive solutions, but this industry has yet to settle into the realities of building enterprise software. And so flow forth of the old ways of meeting new demand in a fast-paced world — the old ways of building a product and solving problems, and ultimately the old misgivings about research:

Chris Avore saw something similar at NASDAQ Design as well, which he writes about in Enabling Design Influence at Scale:

But to [achieve a design- and research-led work culture] requires another problem of scale: preparing the organization to understand and apply design methods typically associated with the design team, and ultimately allowing those teams to solve their business and customer problems by putting those approaches to work.

Advisory understood the value of research — after all, it was built on it. But, the individual product teams were going to need help with understanding and applying design and research methods. We weren’t going to be able to only conduct research and point teams to best practices. Erin Howard on our response:

[We] decided to take another approach, empowering product teams by inviting them to learn how to hack research to gather and include insights into each development cycle. “Teaching teams how to fish” became our mantra, and instead of shame-lighting on gaps in knowledge about our users, we decided to shine a spotlight on key areas product team members could immediately engage in to collect insights.

We’d still take on key research engagements that were expected to deliver near-term value to multiple products, or foundational research likely to be useful to the company in the long run, but anything else, anything that anyone else could do given training and support, we’d provide those things, instead.

Tell them what you’re going to tell them

Different companies have different cultures of information-sharing. Advisory’s is presentation- and meeting-oriented, so with our product manager, Olivia Hayes, we decided on a series of talks we could repeatedly deliver, effectively leading teams through the maturity model.

Chris Avore relates their parallel discovery at NASDAQ Design:

…it’s important to first establish a general shared understanding of the design team’s work and successes.

Create a simple presentation that describes your team’s charter and mission statement, as Russ Unger advocated at MX2015. Include the projects you’ve worked on (and perhaps the ones you do not work on), the methods you use, the locations your team works from (be specific with remote team members too), where your team sits in the larger organization hierarchy, and why design is important to the company.

…creating a common understanding of design responsibilities and expectations across people and teams, designers or otherwise, is the first step to establishing a culture of design and preparing non-designers to follow your lead.

User Research 101 (UXR 101) was the first part of this effort. Originally planned to be a practical talk on the basics of conducting research, I pivoted it to a higher-level, more evangelizing talk, after attending Lean UX NYC 2015.

Lean UX, despite the name, wasn’t really a conference for UX professionals. It was a conference for product managers and development managers to learn about design processes as applied to lean and agile development processes. Two Lean UX NYC 2015 talks are directly referenced in UXR 101:

The Mindful Manager: Developing Equanimity in Leadership by Simon Bennett:

and You don’t know d*@#ck - Continual Discovery & Development w. Dual-Track Scrum by Aaron Sanders:

But, many more influenced the talk as a whole. Talk after talk spoke directly to product managers and development managers, using their vocabulary, assuaging their fears, showing them how good design processes can support product management and development goals, from strategic roadmap decisions to pruning the backlog.

Erin Howard explains why recognizing this was so vital to our success:

We could have continued to flood the inboxes of decision-makers with emails about best practices, or chat-storm links to articles and books written by acclaimed authors who have defined and re-defined processes.

And while these are great resources for people who have the time to ingest or refresh on principles of good UCD, these are also great ways to get overworked product managers and leadership to de-prioritize and ignore your well-intended messages and mistake them for salty/academic/“smart kid” who has no understanding of real-world business.

Without applying your own skill and craft, your empathy towards the roles and responsibilities inside your own environment, you relinquish control of the situation entirely. You are setting yourself up for isolation.

This was the voice that was missing from the first drafts of the UXR 101 talk. As a practical talk, it lacked empathy for the internal users: the product managers and development leads. I was telling them how, but not why; describing features, not benefits. Writing this now, it seems obvious; I’m UX, why wouldn’t I have empathy? But, it can be easy to forget internal stakeholders are users, too.

Our talk outline looked like:

The final ~40m talk has those four key sections, wrapped in a pair of intros and outros:

Here are some highlights for teams looking to do something similar.

Speak their language

As part of establishing our empathy, it was important to put things in the terms product managers and development managers were familiar with.

Throughout 2014, product managers and development leads had been attending workshops led by Jeff Patton, teaching them the “Product Discovery” design sprint process, giving every PM a common, design-related vocabulary for the first time.

This wasn’t the only product development process in use. Lean had been adopted by some of the products.

Some of the most progressive teams worked using dual-track SCRUM.

Rather than provide examples of research processes, I provided examples of benefits, framed in terms of these development processes, and showed where more rigorous design and research practices could directly support aspects of each one.

Understand the business needs

If you look at the benefits I espouse in the talk, they’re not all traditional tasks of a user researcher:

Rather, these are business functions. Throughout 2013 and 2014, we had seen product teams across the organization:

These are familiar problems! Designers and researchers see them all the time. But, there can be a lot of value in forcing things to be explicit. This was the first time anyone had said, “not only are these familiar problems in the industry, but every team here has experienced at least one of them,” in a centralized setting, where everyone could look around and realize they weren’t alone. (And, of course, in the internal presentation of the talk, I was able to cite the actual problems, teams or people affected.)

An individual researcher simply conducting research doesn’t — can’t! — solve any of these problems. But, these problems were getting in the way of teams being able to advance through the design research maturity model, so these were the problems UXR had to solve. It meant UXR couldn’t stay in the “research” box, throwing results over the fence. We couldn’t deliver only reports or mental models or pain points or affinity diagrams. For some teams, we would have to deliver end-to-end processes, training, facilitation, and actionable recommendations.

Use internal stories

To convince teams that integrating research processes was safe and well-established, not just in industry at large, but within the company, I focused on actual internal success stories from the past 12-24 months, when user researchers had existed, but not yet been centralized. Even though I believed there could be many more theoretical benefits than what the researchers and designers had proven out, focusing only on real successes gave an incomparable weight to the talk: only points with success stories were included, and so every point had a success story.

Furthermore, while the public version of the talk doesn’t get into specifics, the internal version absolutely used product names, team names, department names, and even cited individuals. And, when those people were in the audience? They absolutely were looked at to validate what I was saying.

Here’s that model again:

From slide 44, this was a “Laggard” team, only looking for answers after something went wrong, and not having enough empathy for the user to not blame them:

Or, the user perceives the data in the product to be inaccurate, regardless of whether the math is correct, and so your product utilization plummets.

This was an “Early” to “Progressing” team, committed to making only new mistakes:

…and understanding how users think and feel about their data, or its collection processes, or its attribution, means we can design and build reassurances about that provenance into the system from the beginning. And there are products where that’s been very successful.

From slide 59, this project was “Progressing” to “Mature” in terms of scope, and delivered a lot of useful insights which I can’t talk at all about, as they’re still being cited and utilized, 2+ years on:

A new product development team wanted to come up with new product ideas to test. There’s a lot of value in having different, competing ideas to test at that early stage of the process. No-one’s built anything, so there’s very little opportunity-cost in throwing one of them away if it doesn’t perform well in testing.

Chris Avore mentions that it’s possible to find yourself in a “bingo board-like organization,” and this next team fit that. The intended scope was “Progressing,” but the attitude, purpose, staffing, audience, and governance were all squarely “Laggard.” We learned so much about unspoken company culture during that project:

In another example, another product had already launched to alpha customers when I interviewed the subject-matter experts on the product and reviewed the feedback that had already been collected. Instead of recommending redesigning small portions of it, or discovering the “one feature” that was missing, we came up with two or three similar, related, but ultimately different products that that end-user might use, but there wasn’t really a path to there from where the product currently was. Now, this didn’t kill the product — its sunset was already in the works — but it was another data point in support of the sunset.

From slide 60, again, we focused on a research-related evaluative outcome, rather than a design-related one, like a usability test, as most teams have seen value from usability tests when they conduct them. They were “Progressing” to “Mature:”

One particular product team is a good example, here. They had us review their app and their interpretation of the research they used. Now, at the time, there was a big push to focus on a large group of potential users. But, for their particular use case, we felt the research was actually saying a particular subgroup should be the target. Now, of course, had they kept focusing on that larger user group, they would have gotten the subgroup as part of it. But, by having us come in and double-check their work, we gave them a nudge in a direction that we think is going to result in a product that is more likely to be used by its narrower, more specialized audience.

No gatekeepers

While our recent re-org hopes to make this less of a problem today, at the end of 2014, Advisory was still pretty siloed, and it could be very time-consuming to find people in other departments and offices that knew about the space you were interested in, let alone get their time to brief you about it.

In the internal presentation, this was an internal address that went to a shared inbox, where any of the researchers were empowered to respond, provide answers, references, or even take on work directly if they didn’t feel it would impact their business priorities. As a way to reduce friction and give an unconvinced PM no excuse not to reach out to you, it was very successful.

Immediate takeaways

The first delivery of the talk dove right into “Tell them what you’re going to tell them.” The introduction slides were added later, to provide immediate takeaways for motivated product teams.

This list is the (very) short version of a standard set of incremental recommendations that we provided to all interested product teams, providing an outline for increasing rigor in all their design and research processes, and slowly involving all related departments, including sales, support, and others.

Chris Avore’s September 2015 post, Achieve More Research, More Frequently, includes many of the same things our standard internal recommendations did:

This emphasizes for me that enterprise UX is the same all over, which means the more enterprises that publish about their experiences and their solutions, the sooner we can all work on new problems.

Look and feel

Along with the single point of contact, another way we subtly suggested we weren’t playing by the old rules and just carving out turf, was with the presentation itself.

If you poke around on Advisory.com, you’ll see examples of the corporate-branded slide decks and reports, which are super-traditional. Lots of words on slides, because slides were expected to be distributed and read after the fact, etc. Internal presentations tended to go the same way.

The UXR 101 talk has very few slides where the narration includes any of the slide content, so no-one kept their phones out or laptops on once I started clicking through them at a fast clip. Olivia Hayes rightly insisted we didn’t offer a recording; you had to see it live, and I delivered it every month.

The presentation theme itself, designed by Rondal Scott, was super off-brand, enough so that I had to remove the Advisory branding for the public release. But, all of these things add up, and help convince people you’re genuinely doing something in a new way.

References

I mentioned two particular Lean UX NYC 2015 talks earlier:

If you’re in the enterprise space and need to relate to and empathize with product managers and development managers, you could do worse things with your time than watching all of the videos Will Evans has posted from it.

The PM quotes early in the talk were chosen by Olivia Hayes and Erin Howard, and came from:

Other PM-related essays that were cited in earlier versions of the talk included:

The books were:

Desirée Sy’s original dual-track SCRUM paper is here:

If they had existed when we were doing this work two years ago, Chris Avore’s essays would have been very useful references, and I highly recommend them for enterprise teams today, as we saw all of these same things come up at Advisory:

Thanks

Olivia Hayes, Erin Howard, and Chris Allison did most of the editing and review for the deck, and I am indebted to them for it.

Dorian Freeman, Susan Farrell, and Dan Turner, from the Designer Hangout Slack, offered feedback on this essay, and I sincerely appreciate it.

If you have questions, my contact information is on the home page, and I’m @vitorio on the Designer Hangout Slack in #topic_enterprise.

It’s July 7, 2016, thanks for your attention.