Measuring What Matters

Kristin Wisnewski
VP, CIO Design, IBM

From the DesignOps Summit website:
Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

She is 1% of that 3% working in the Office of the CIO.

She started at IBM as an intern nearly 20 years ago.  She worked with so many amazing people, many of whom are serving as trusted advisors to her now.   What she is doing now feels less like a job, and more like what she is here to do in life.

There are more IBMers than people living in Iceland; the population is roughly the number of people living in Cleveland Ohio – it’s a lot!   Just over half of IBMers have been there for less than five years – she and her team are an anomaly.  The newer employees have high exceptions and low tolerance for things that aren’t great.  And another 30% of employees are either working remotely or at a client site.  

The mission for the CIO is to conitinously improve the work experience of employees. If their responsibility is to create a productive environment, they need to have empathy.  The state of IT is a reflection of how IBM thinks and feels about its employees.  

They have a legacy tool stack as well as new solutions. GIven that environment, they have to be measured and precise about how they support their peers – be an effective shared service for them. Instead of Maslow’s Hierarchy of Needs, they have created the IT Pyramid of Pain:

She was a UX practitioner for 14 years, and then moved into management.  She has a small team, a couple of tools, and she used to keep a whiteboard of activities in her office.  

Now they have a bigger charge to grow the team to support employees at IBM. Today it looks like this:

We are at a poitn where we are impacting the whole work experience. Not just tools, but tone and culture. And also the physical environment as well. They are trying to influence what it’s like to work at IBM,

They look at everything they are doing – they make look at these metrics daily, and at the end of each month they publish these reports:

The team is 120 designers, another 25 coaches. They like to say they are here to make people and teams better. They also have a content and communications team. They are trying to send less, but make it more impactful.

The research team is about 20-ish people. This past year they have talked with 80K employees, and it’s a reflection of the fact that people care, and they want to influence their work experience.

From a communications perspective, they track a number of variables, they also do A/B testing on subject lines and more.

In a more reactive way, they support projects. They are now starting to look holistically and comprehensively based on their past work. Using a more proactive approach, they can see trends, avoid duplicative studies, and more. They are also able to injet their own ideas and dreams into these new projects, while at the same time resulting in savings for the company.

Voice of the Employee Program (VOTE) is a way they get feedback from employees. Rather than a giant survey which relies on retrospective on tools people may have not used in awhile. They can ask questions about NPS, ease and capability. What were you attempting to do, were you able to do it? They are now getting 4K survey responses a month into their repository, which helps to fuel those proactive efforts.

They have also created a Library of User Experience (LUX). By having a repository of past reseach, it helps to avoid pushback for doing discovery / research at the front end of projects. This data is a growing set, it will get better and better over time. At the outset it was very private – now all that data is available, because they have found that it helps people commit to improve.

Now they are tying this data back to performane reviews for the people working on those products – not specific scores, but to see trending in the right direction. NPS is a good place to start, but it’s not the be-all, end-all. Other metrics include Goal Completion % and Ease and Capabilites. They also have a scorecard per solution with all those metrics, and tracking by quarter. All the qualitative feedback is collected as well.

The library is organized, all the data is tagged. They are now looking to get to predictive measurement – to get out in front of what users are looking for. Their team is looking to take friction out of the system for employees, so they can focus on delivering value for IBM and clients.

Their full suite of services:

Their Employee Directory hadn’t been updated in 12 years, and it was getting thousands of lookups every day. Now they have an experience that is much more Google-y. They have also overhauled their Help system, which works across platforms and devices. And the corporate intranet has also been significantly updated. They are trying to make it a bit more personalized, and offer dark mode, people! 🙂

They also developed a Workstation Refresh Program. In the past, employees got a new machine every four years. But only 30% of people were taking IBM up on that offer. Some people want new technology, others don’t want the disruption to their work. What if they made it an on demand program? At the same time, they used AI to identify people who were at risk of critical failure if they didn’t upgrade. Finance did not like it at all – but now it’s the new policy.

But measurement is not without it’s challenges. Forcing numbers and process on people that don’t want it. One of her data analysts said something that really resonated – about the importance of using the data to help make the right decision, rather than inform a decision you’ve already made.

One team used a version of NPS in color, which really influences how people respond to the survey. She had significant concerns about the impact on the results, so she replace their colored survey with the standard one, and quarantined their data. Although it caused some tension, it turned out to be a beautiful A/B test:

Their DesignOps Journey:

• Stage 2 – in the early stages of DesignOps, they had 150 Trello boards, none of them speaking with each other.
• Stage 3 – now it’s more evident where they are and where they are going; getting more comfortable embracing the chaos.
• Stage 4 – we know it will be better, but we’re not sure what shape it will take.

Measure what matters – even that small circle can make a big impact.