Predicting The ROI Of eLearning

Data-Driven Decision Making: What Is The ROI Of eLearning?

This article is part of a series on tracking the right metrics using the right methodology to visually display and quantify the investment in learning and prove ROI. What you are about to read is a fable. The company, AshCom, is fictional, but the learning challenges faced by Kathryn, AshCom’s CLO, and her team are real and commonly shared by learning teams in large organizations. It is our hope that you will be able to connect with the characters, their challenges, and the solutions they discover. We also invite you to read the first eBook in the series.

A Whirlwind Month

Kathryn inhaled deeply. She held it for just a second before slowly exhaling. As she did, her mind traveled back over the previous weeks. It had been a whirlwind. She sipped her tea and realized that while some of her thoughts had settled, some still swirled around in her mind.

The last month had been more stressful for Kathryn than any other time in her professional career as a learning leader. Her role as the Chief Learning Officer at AshCom had been filled with challenges that tended to give her energy. Her learning team was made up of smart and driven people. She worked closely with them to meet every challenge that had come their way.

Together, they figured out how to develop a completely new learning system when AshCom acquired a competitor. When their CFO wanted to increase the financial literacy of his managers, Kathryn and her team built a true game that was successful in every way and was one of the most popular learning experiences they’d ever built. Her team built its own mobile app to train the sales force. When a significant portion of AshCom transitioned to working from home, the learning team made the transition. Her team rose to the occasion each time.

The last month was different. AshCom had lost money in the previous quarter, something that had never happened before. In the next two months, things had gotten worse, not better. The poor financial performance was setting off alarm bells throughout the company. Wages were frozen. Any new hires now needed to be approved by Kurtis, the CFO. Tensions were high throughout the entire organization.

Defending Every Dollar

Kurtis met with Kathryn and told her that every department was being asked to “Defend the Spend,” their expression for looking at every expenditure in detail. Every leader needed to defend every dollar they were spending and link it to how it would create value for AshCom and improve its financial performance.

Kathryn normally took a team approach in the face of challenges. That would not work now. She needed guidance from someone who had experience in helping learning teams determine the return on investment building learning experiences. Kathryn turned to Amy, a consultant who had spent a lot of time helping Kathryn think through strategy. Amy worked with dozens of large companies and organizations throughout the United States. If anyone could give the right advice, it would be Amy.

Kathryn and Amy met several times. They discussed the threat to the learning team. Amy had seen this pattern repeated in numerous companies that were struggling financially. The first budget cuts and layoffs came to the learning team. Kathryn knew this too. Amy was direct and told Kathryn that she owned this problem, and it was up to her to overcome it.

Amy laid out a path forward that helped Kathryn decide what to measure, how to measure it, and how to make her report visible through a dashboard that would rival those of AshCom’s sales, operations, and human resource teams.

To get a visual overview of the learning impact, they needed to track the right metrics and use the right methodology. Amy recommended a product called The Learning Scorecard, which was built by the learning experts at MindSpring.

Kathryn’s stress level retreated as she realized she could actually accomplish what she was being asked to do. Amy was giving her confidence.

Kathryn knew her own mind. Stress interfered with her ability to be creative and think about the bigger picture. Her style was to lower her shoulders and attack the problem in front of her. This helped her get done what needed to be done, but it limited her time spent thinking strategically.

The Learning Scorecard

The Learning Scorecard software and dashboards gave Kathryn confidence that the right solution was coming into focus.

She now had time to think.

She decided to rent a cottage on Gull Lake, which was about an hour north of Minneapolis. The weather was perfect for sitting on the porch and watching the boats go by. She knew that some peace and quiet would not only restore her soul but would also give her time to reflect. It was often in serene settings that she had her best ideas. Not everyone would be comfortable sitting alone for three days. This did not bother Kathryn.

After making sure everything was in order with her family, Kathryn left while it was still dark, made the hour drive, and settled into her new temporary home. She made a cup of tea and went out on the porch just as the lake was waking up. She watched the early morning fishermen come in while the powerboats and jet skis were heading out on the lake. The porch had a large swing hung from the ceiling by heavy ropes. It was large enough to hold a twin-sized mattress so she could sit or lie down if she wanted to. The day was perfect.

Near the top of the legal pad that almost always accompanied her, Kathryn wrote, “The Learning Scorecard.” She and Amy had spent so much time implementing it that she hadn’t had time to think about what else might be possible. Now she had three days.

Next, she wrote “Backwards/Forwards.” It occurred to Kathryn that The Learning Scorecard system gave her clarity on what had happened. As she watched a speedboat glide by, she thought that the boat might represent the learning experience and the wake behind the boat was sort of like the impact of learning. What she was measuring would be what happened after something else happened.

How Was Behavior Changed Through eLearning?

Kathryn did not discount the value of a clear view of what had happened. This is what Kurtis, the CFO of AshCom, was asking for. He wanted to know how behavior was changed through learning. If he could get a quantifiable sense of that, he could compute performance improvement due to learning.

With some easy math that involved the cost of building the learning, he would be able to get to a return-on-investment number. He was already able to do this by adding new employees, changing an operating procedure, or buying new equipment. He would now be able to run similar numbers for learning.

Kathryn started thinking about what sort of patterns would emerge over time. If she and her team could see clearly backward, wasn’t it logical that they would have more clarity looking forward? She wrote another word on her notepad: “Causality.”

In time, Kathryn and her team would be able to see which kinds of learning experiences produced the greatest return on investment. She might even be able to predict the ROI when proposing new learning experiences to the executive team.

Data-Driven Decision Making: Asking The Important Questions

It occurred to her that this was not the only usable insight to be gained. She thought about the learning team and causality. She started to write some questions:

1. If her learning team gave a learning experience high marks, did that necessarily translate into stronger learner reaction?
2. If learners reacted well to an experience, did that mean they performed better in that course?
3. If learners liked an experience and they did well in the learning experience, did that mean there would be a higher level of behavior change?

More questions rolled through her mind. Was there some modality of learning that learners liked more than others? Could she track learning that was reinforced after the fact compared to learning that was not?

Then she thought about learners. What if different departments tended to react better or worse to certain kinds of learning experiences? What if salespeople preferred a heavier emphasis on gamification while machine operators preferred microlearning in short bursts that they needed in a moment? Maybe managers preferred scenarios that gave them the opportunity to make choices and see outcomes in more real-world situations?

It would take some time before these patterns emerged, but Kathryn knew that she and her team could start looking for them immediately. Every learning experience would have performance data behind it that would guide their decisions moving forward.

Kathryn decided it was time for a break. She took a short swim in the lake, changed, and made herself lunch. Seafood salad in lettuce wraps and more tea.

When back on the swing, she picked up her legal pad, flipped the page, and wrote, “XR” at the top. For more than a year, her learning team had been bringing up augmented and virtual reality. Some members of the team had spent considerable time reading about and playing with virtual reality. One claimed to be addicted to the augmented game Pokémon Go and spent weekends chasing them.

With the financial challenges of AshCom, now was not the right time to put in a request to expand their learning choices into augmented or virtual reality. She knew it would be added expense, at least in the early stages. But Kathryn also knew that the time was coming when her team would need the technology. She and her team would have to become more familiar with it.

Experimenting With Both Realities

The Learning Scorecard would give her team the opportunity to experiment with both realities and compare the results to more traditional forms of learning. Instead of her asking for an increased budget based on new and very cool technology, Kathryn would actually be able to demonstrate learner reaction, learning, and behavioral change. Even more importantly, she would be able to make an ROI case for the technology. She would start small and prove the concept before she asked for a larger financial commitment to the technology.

Kathryn flipped the page. Time for a new topic. She wondered how this kind of measurement would change her learning team. Would this be threatening to them, or would they see it as an opportunity? Being measured and held accountable for outcomes might not be on the top of everyone’s list.

Kathryn thought through each person on her learning team and decided that all of them would welcome this structure because they all had a passion for contributing to people’s growth in knowledge, skills, and opportunities. A few of them might begin to think more like business leaders, moving past intuition and into data-driven decision making.

This was enough for her first day. Kathryn had a list of questions to think about that would occupy the remainder of her time. She was also sure other questions would arise as she sought answers to each one she currently had. She decided to draft an answer to each question and then spend some time with Amy to get her reaction. Sooner rather than later, she would have to have an open conversation with her learning team.

It was time to start thinking about dinner.

Conclusion

Download the eBook The Learning Scorecard: How To Measure L&D Impact And Prove ROI to delve into the data and discover which key metrics your L&D team should consider. You can also join the webinar to discover a completely new approach to measuring ROI.

Dear Reader, if you would like to see a demo of MindSpring’s Learning Scorecard, please click here to schedule a time convenient for you and the learning experts at MindSpring will be happy to walk you through it.