.jpg?width=1920&name=008-LITES-MRUSTICI-web%20(1).jpg "008-LITES-MRUSTICI-web (1).jpg")
Mike Rustici, Watershed LRS with Zack Parnell, ITI
MANDY HENRY: Welcome to the LITES Podcast. It’s Leadership in Industrial Technology, Education, and Safety where we talk to thought-leaders in construction and heavy industries in this exciting time of innovation. I’m Mandy Henry, Communities Manager at ITI, and in this episode, we got the chance to talk to Mike Rustici, Founder and CEO of Watershed LRS about SCORM and xAPI and how these programs are helping other companies build their Learning Management Systems. Thanks for tuning in.
ZACK PARNELL: Mike, thanks for joining us. Can you tell us a little bit about yourself?
MIKE RUSTICI: Yeah, thanks for having me. Yes man. I've had a long and distinguished career dealing with obscure technologies that seem to have a pretty big impact on the industry. So I started as a software developer many, many years ago and eventually started my own company called Rustici Software because it was just me in the spare bedroom and had this idea that I wanted to go build a company filled with really, really smart people because I thought if I could attract and retain really smart people that you know, it'd be a great company to work at and I would really enjoy life and would find some interesting problems to work on. I set out to go create that and very quickly found a niche working with this obscure standard called SCORM. SCORM is the way that eLearning courses work with LMSs. And that turned into a nice little niche and we had SCORM.com and we quickly became the go to destination for all things SCORM. It was enough of a niche that we could create a 40 or 50-person company out of it, but not a big enough problem that, you know, Microsoft or Google or Oracle ever wanted to come in and start doing that stuff. I was able to be very successful in carving out that company that I wanted to build filled with really great talented people that I enjoyed working with each and every day.
That led into the year 2012, an opportunity to partner with the Government Research Lab and the Department of Defense called ADL. They're the people who created SCORM originally, and we had the opportunity to go partner with them to create the next version of SCORM, it was something we called Tin Can. The project Tin Can and the Tin Can API. It's a very beginning that then got renamed to the Experience API or xAPI. I had an opportunity to go work again with another of these very, very kind of niche standards, but that is really, really kind of important.
When we did that, when we did Tin Can and created xAPI, we knew we'd created something kind of pretty big. We knew that standards had a big impact on the industry. We started up a new research group called Watershed to go figure out what are the impacts going to be of this new way for learning systems to work together. And we're very fortunate with that. You know, our first customer was Google and our next one was AT&T, and we signed up a dozen or so of these really high-profile companies who wanted to do really innovative things with learning to go experiment with what this new technology could do.
In 2016 and actually we spun that company off into its own entity, which I'm currently running. Well, a business partner continued to run Rustici Software as part of a larger group that wound up acquiring it back then. Right now, I've made my way into being the CEO of a startup company again. With Watershed we're trying to go change the face of corporate learning by introducing the concept of learning analytics, the concept of using data to drive the culture of continuous improvement and to be able to connect the work that we're doing and learning to organizational outcomes and show whether or not we're in fact having an impact.
PARNELL: Very cool. Very cool. Could you help us understand, a lot of listeners probably don't know what SCORM even is. So, for the layman, what is SCORM?
BUSTICI: SCORM stands for the Shareable Content Object Reference Model, but you don't need to remember that. What you need to remember is that SCORM is the way that tools in the eLearning industry work together. It's a standard. It's a defined way of writing software so that it can work with other software.
Let me give you an analogy to explain why you care about standards and why they're so important and so impactful. The best way to describe a standard is to use an analogy to another standard, one we're all very familiar with would be USB, the plug we put into our computer. I like to say that standards are important because standards define and confine our marketplace. In the world of USB, if you want to sell a product that plugs into a computer, you have to work with USB because that's the standard. If you don't work with that standard, you'd have to go customize your product for each and every computer you want it to go plug into, and that's not scalable. You can't sell that on the mass market. Well that USB cable has constraints, whether that constraint is data transfer rate or power availability. All of those constraints are things that those products need to live within. If you need more power or more data transfer than that cable provides, you can't work with the USB and you can't sell your product on the mass market. The standard confines the products on the market, the standards define and confine the marketplace.
Well in our world of corporate learning, SCORM has been the standard that we've had for a very long time now. SCORM was created in the year 2000 and I'm a little bit embarrassed to admit that was really, really excited when I first read it. That just goes to show you how geeky I am because SCORM was brilliant for its time. There was really forward looking and solved a lot of great problems and really enabled the eLearning industry to blossom and now that we had a standard that allowed us to go create a piece of eLearning content once and have it work in any learning management system that was out there., it allowed for these new offering tools to pop up and for these libraries of content to pop up and really just enabled that entire industry. But, it's 2018 now and remember SCORM was created in the year 2000. That means we're living in an industry that's been defined and confined by an 18-year-old model. You know, it was defined and confined by Y2K.
That was the era we were living in when SCORM was created. It doesn't really reflect your mobile and social and Cloud and all of the new things that have come along since then. Which is why I'm really excited about the possibility of this new standard xAPI that is starting to be widely adopted throughout the industry. xAPI you can think of as kind of the modernization of the learning backbone, the modernization of the learning systems that we can have in our industry. This allows us to take advantage of all those things that were envisioned when SCORM was created with mobile, social, Cloud, etc. It allows us to not be confined by what we can put into a learning management system, but really, take advantage of whatever technology might be most appropriate for delivering learning: AR, VR, all these cool new things. Or something as simple as just incorporating more real-world coaching and mentoring events or tracking the fact that we might read a book or an article or an informal bit of content, whether that's online or offline., the real world or the virtual world.
xAPI allows us to bring him a lot more different types of learning experiences into our ecosystems right now. But the big thing the xAPI really does is it unlocks a lot of data. I like to ask people, when I'm speaking about the learning industry, “What percentage of what you've learned in your lifetime has come from an eLearning course and an LMS?” It’s a really, really tiny fraction. xAPI allows us to tap into data and information about all of the other places where we are learning. It allows us to tap into that data and bring it together with data about employee behaviors and employee performance that we can go now see if the training and learning programs we're delivering to our employees are actually effective at changing those behaviors and improving that performance.
PARNELL: Great overview Mike. We'd love to learn a lot more about ADL and why the Advanced Distributed Learning Group of the government got involved with xAPI. What was the need? What was the impetus?
RUSTICI: Yeah, so I think for that answer, we really should go all the way back to SCORM where they first got together and started standardizing eLearning back around the year 2000. First of all, you have to understand ADL is part of the Department of Defense. The Department of Defense is one of, if not the largest training organization in the world. A big part of what they’re doing is training and developing people in a very rigorous and intense way, with a huge priority. Especially these days on being agile. Responding to new threats that are constantly popping up and new ways of warfighting that quite frankly didn't really exist for many years ago or weren't as commonplace, I should say. at least. They saw the promise of eLearning back in the day as this way to accelerate their ability to train people better, faster, smarter, more intelligently. That's where we're SCORM came from.
SCORM came from the need for the department of defense to be able to buy an eLearning course that would work at all of their different military bases. Prior to SCORM, every base had their own different LMS and you could buy a course in one place and it wouldn’t work in another place. There was a simple kind of procurement efficiency that really was actually what really drove SCORM home. I could buy this once and reuse it several times. In addition to kind of this vision they had for creating this really adaptive intelligent tutoring system based on reusable assets.
So, they kind of became the de facto curator of an industry standard and know that that's not necessarily the best role for the government and they've tried several times to kind of pass that responsibility off to the industry. But it was very difficult for some very legal bureaucratic reasons with the way they created SCORM in the very early days. xAPI, their motivation for doing that, it was essentially that SCORM was getting a little long in the tooth. It had been a dozen years since SCORM was created and it was quite obvious that it's age was beginning to show, and the DOD wanted to be able to do things that went beyond the confines of what SCORM could do with their really old model. So, they initiated a research project where they looked at a few different ways of creating this kind of next generation learning interoperability standard and one of the things that they did was to give my old company, Rustici Software, a research grant to go have us figure out what we think the next generations look like, and we really embraced that. We really embraced an open community approach to doing that.
We involved, I think it was 300 or 400 different industry participants to gather requirements and use cases about what this next generation would need to support and came back and synthesize those onto a proposal, then kind of sent that back out to the industry to validate that. Hats off to ADL, they learned from their lessons with SCORM and they released xAPI completely open source. It's now a truly an industry wide effort run by the community who are constantly coming together to validate it, then update the spec and ensure that it is in fact working. There's actually a team effort right now to transition the formal ongoing work of xAPI out of ADL over to the IEE, which is kind of an open standards body that really anybody can participate in, so it's not locked up in just this governmental organization.
PARNELL: Awesome. I was wondering if that was the case with how it was going to be transitioned out into the public. So that's cool. Mike, a lot of listeners work for heavy industrials and construction companies. The workforces there are doing a lot of hard skill works that we call it as opposed to soft skills. So, welders and boilermakers, pipefitters, crane operators, and craftsman of all sorts. So, for instance, in a crane operator’s training and development plan, they have several things they're doing. They're logging operator hours at all times when they're in cranes, and there even logging hours, obviously per cranes. So, if they're in a Boom Truck one day and then a Rough Terrain for two weeks the following week, but additionally they're taking prep courses and exams, they have to recertify every five years. All that stuff. So, what sort of standards, how does xAPI work and what's in place to capture those sort of learning objectives?
RUSTICI: Yeah, great question. That's exactly what xAPI was designed for, to recognize that learning is going to happen in many different things. You know, you mentioned, real world operation of machinery and have courses and exams and perhaps things happening in a simulator for some of that heavy machinery.
xAPI allows you to capture data no matter where it came from, and in a very flexible format. The xAPI standard captures a series of statements about learning experiences of the form: I did this act or an object. It could be: Mike operated the crane, Mike finished his course, Mike scored 85 percent on his exam. It's very, very, very flexible both in the terms of the types of things that can be tracked and the granularity with which things can be tracked.
We run a crane simulator, for instance, you could track just how far I raised or lowered the boom and how tightly I set the break or whatever it might be in that particular world. I don't know if those are valid things are not in the world of crane operating.
PARNELL: That's a great point. I mean, we're interested in tracking. A lot of our customers want: Where are my operator’s eyes? Damage to equipment is a big key, it's whether you were setting a load down, imagine setting a 50,000 pound a load or a piece of equipment down on the ground really hard, you could actually destroy a piece of equipment. There's so many unique little nuance things and what's neat about simulation, as you probably know, the simulation engine can capture that information. So that's pretty cool to realize.
RUSTICI: Yeah, and so if the simulation engine has starting to capture that, it’s a way to feed that into your learning record store, which is the central repository of your xAPI data, and bring that together with data about actual damage to equipment in the field. We could go look and say “ Of all of the people that we've ran through the simulator, whose eyes were always in the right spot, their damage to equipment rate was 30 percent less than the average” Or things like that. We can really go dive into understanding which factors in the training lead to the greatest impact on the business.
But to do that, to be able to capture such a wide variety of data is both a blessing and a curse. It's great. The blessing side is easy. It's great to be able to capture and understand so much, but it also requires some discipline and almost to have a series of best practices and profiles to allow for all of that data to be useful to other places and in other systems.
So, for instance there's a whole community around figuring out how to use xAPI to capture simulation data. There's another one for using xAPI, the capture video data, another one for using xAPI to capture your learning data. So xAPIs have this blanket umbrella that is over a lot of these little communities trying to figure out the best way to use it for specific use cases.
PARNELL: So, Mike, from our experience, I'm sure you're aware that a lot of companies already have existing learning management systems. So, I suppose the question is how do those start utilizing the benefits of xAPI?
RUSTICI: Yeah, that's a great question and I really think that goes back to the overall learning ecosystem that we have in our organizations. I talked about standards defining and confining our tools. I think our tools have defined and confined our imagination as well and we've been for the longest time, for over a decade, is confined to this model of putting everything into the LMS because that was really the only tool that we had. But the next generation learning ecosystem I think is much broader than just the formal learning that happens in an LMS. It's going to happen in formal ways, through videos, micro learning, gamification, VR simulations, and all of these different things. So, when it comes to getting started with xAPI, we often encourage people to find applications of some of this new technology that they want to use and then capture that to get the analytics out of that information. You need something called a Learning Record Store or an LRS. Self-promotion alert, that is what watershed does, and we'd love to sell you a subscription to ours, but we also have a free version called Watershed Essentials that you can use. Several of our competitors have really good tools out there. Some of them even open source that you can download and use to get started.
Find something new and interesting and different that you want to track and understand that didn't necessarily fit into the LMS or the traditional training bucket. We're seeing a lot of really, really great xAPI adoption starting to happen. It's mostly amongst the smaller vendors who are bringing new and innovative tools to the market, the people who don't live inside the SCORM box. If you look at a lot of the traditional eLearning tools and the LMSs, they've become very successful large companies by living in the box we've all lived in for the last 20 years and xAPI is a very kind of different way of looking at the world.
For instance, LMSs for the last 18 years have kind of taken the approach that all learning should happen within me. If you want to do learning, it should happen within me because that's the place you go to learn. But that's not the way the world works. I don't care how great any one system is. Learning isn't intrinsically human activity that can, will and should happen in many different places and in many different ways. The xAPI model just kind of flips it on its head and embraces and says “Let learning happen in the right moment of need in the right place you need using the right modality. Don't force people to come into one place and we'll still let you track that and have information about that as required for your business and in this learning record store that sits on the back end.” That's a very, very disruptive type of a model for an LMS.
When we are seeing some of the larger LMSs adopt xAPI, they're doing one of two things. They're either saying, “We'll use xAPI in a very limited way to do the same things SCORM could do before or we'll use it Just to track the data inside of our system, which is kind of only marginally interesting.” The other thing we're starting to see a lot of them do is to be not a consumer of xAPI statements, but a publisher of xAPI statements to recognize the fact that the LMS is just one component of the modern learning ecosystem and to expose the data that is in their system out to other systems to play nicely as part of that group and we really see the role of the LMS within the corporate learning ecosystem shifting pretty dramatically. They're not going away. LMSs serve a vital function of managing formal learning, dealing with compliance, delivering e-learning, managing classrooms, etc. But what we do see as their primacy in the learning ecosystem is shifting and diminishing to be just one component of the modern learning ecosystem that also includes informal learning activities and performance support and mobile applications, etc.
PARNELL: Fantastic. So, Mike, we were chatting earlier about how ITI and United Rentals and Serious Labs were building a virtual reality crane and equipment simulators, and one of the big things we're trying to do is tie specific training plans to learning outcomes, which equates hopefully into better performance outcomes for the business. Do you have some good examples of how xAPI has helped organizations do this more efficiently?
RUSTICI: Yeah, absolutely. I'm thrilled to hear that you're working in that area. VR has so much promise. As with any new technology to prove that it's more than just a toy, you need to show that it is actually effective, and I think that shifting the mindset of corporate LND to be more focused on demonstrating that we are in fact having an impact will be the most profound change in our industry over the next 5 to 10 years. I think it's going to become something that is just expected of us as it is the rest of the business. Yet people talk about that a lot, but sometimes it's really kind of hard to know how to approach that or what works and how others are going about that.
So, let me give you a couple of examples from of our customers who have just seen do some really amazing things. The first example I'd like to share comes from a company called Applied Industrial who is a distributor of industrial parts and they have a couple thousand warehouses throughout North America, Australia, maybe a couple of other places where they have a warehouse manager who has been a blue collar worker who has risen up through the ranks from the warehouse to become the manager of the store. They're not an MBA, they don't have a ton of finance knowledge and things like that.
Well, that was problematic for Applied Industrial because they recently procured a new SAP system that they spent $80 something million on and that was really going to give them a lot of great insights into the operation of their warehouses and how efficiently each of those warehouses was operating. But to understand the data coming out of that SAP system and to be able to act on it required kind of a basic understanding of some basic business finance concepts that a lot of these store managers had never been exposed to before.
So, they put in place a basic business finance course that they distributed to a lot of their store managers and they wanted to measure whether or not that was going to have an effect. So, what they did is they started by measuring each of the learner’s competencies across some key areas of basic business finance. They measured them before and after the delivery of this basic business finance course out to everybody, but they were able to associate key performance indicators or KPIs from each of the warehouses that were tied to each of these knowledge competencies and they were able to get very specific data about those key performance indicators from their SAP system.
What they were able to do is they were able to use xAPI to bring all this data together, all of the data about the competencies assessed ahead of time and afterwards bring together key data about the actual participation and the usage of the training course, but then also bring in data about those KPIs. They were able to go in, they had enough of a data set to go in and demonstrate that there was a very clear causation of that course increasing the performance on those KPIs as the knowledge competency increased. We were able to go in and demonstrate a seven-figure impact on the bottom line by knowing how much that increase in knowledge competency increased the KPIs. That's one great example that ties in training and learning specifically to very targeted business outcomes and to dollars and cents at the end of the day.
My favorite example comes from a hospital chain called MedStar. MedStar is a hospital chain based out of Washington DC and they have a couple dozen hospitals. But they were below benchmark in something called code blue response time. Code blue is, if you're not from a medical profession, when your heart stops, you're literally turning blue and about to die. In that situation, response time, how quickly you can get paddles to the chest is one of, if not the most significant predictors of life or death. MedStar was below benchmark in that response time and wanted to improve that. What they did was they put together a blended learning program to go and train their staff up where they started off with a blended learning curriculum in the learning management system and some classroom training and stuff. Then they also had a simulation on a mobile app that allowed for their staff to kind of practice the best practices and responding to a code blue. Then they would couple that with real world simulations. They were called mock code blue, and they would have performance observers watch how the various staff reacted to the code blue. They would record all of that data with xAPI through a tool from a vendor called Zappy Apps that produces some really great innovative next generation learning system tools.
Then what they did, they were able to go look at that real-world performance data. In this case, one of the things that they noticed was that one of the biggest delays of time was the early stage of identifying a leader and they were able to go say, “Oh, that's one thing that's taking a whole lot of time.” Then they were able to go back and cycle that into the training programs, go improve the training on that specific one subject. They can kind of continue to iterate and iterate and iterate to go slice off even more and more seconds, milliseconds off of that average response time. It isn't about dollars and cents, but it goes directly to saving lives. I'm really excited to have just been able to participate in that and maybe we shaved a couple of seconds off and that saved a couple of lives and that's a really fun impact to have been able to talk about that and participate in.
PARNELL: Yeah, absolutely. Especially when you have a large sample sizes are large population like that, and you're also tying in performance evaluation in the same system. It seems like you can draw much clearer lines between for what value you're creating. Right?
RUSTICI: Absolutely. I think that's the big game changer. The xAPI allows us to capture both of those types of data in the same format, in the same system and really be able to make those connections.
PARNELL: Yeah. Would you say the leading LRS companies or even human resource management systems actually have to tie in performance evaluations a bit in a seamless way? Which historically, LMSs don't do right? They're just delivering content tracking completions. Are a lot of companies that are involved with xAPI, are they kind of getting into the performance management side of it as well?
RUSTICI: Yes and no. A lot of them will take performance data from wherever it is, and they'll let a best of breed performance management system capture that data and then bring it over into the learning record store. Performance management is a pretty broadly interpreted word. I think there's a very specific type of performance management system which is your annual review and your promotion assessments and things like that. But performance is much bigger than that. If you're looking at a general talent development sense, then yes, those performance reviews, retention, promotion, a lot of the things you're looking at, but performance could also be your incident response system or going back to that crane analogy or whatever system tracks damage based on improper equipment usage, that's really going to be your performance review system for crane operator training or safety incidents, compliance incidents.
If you're doing sales training, it might be the CRM or if you're doing customer support training, it might be a customer satisfaction rating system. That data can really come from a lot of different places. So, you don't see any kind of one system trying to be the system that captures all of it. The system is really taking an approach to being open to taking data from wherever it can be accessed.
PARNELL: Yeah. It's interesting. This sounds like an opportunity for timing and several systems together and just sifting through data is a tough thing for most people to do. So, it sounds like an opportunity for machine learning. Actually, machine learning is getting into obviously personal assistance in a big way, but even I've heard of different applications for coaching and understanding how people are performing. Siri has become been a mentor or a coach, as opposed to just ordering a pizza or something. Very interesting.
RUSTICI: I think there's a big opportunity to start tapping into all of this data that we have.
PARNELL: I have one other unique question for you that might round us out well. I had this funny idea. I'm a lifelong learner. ITI is a learning company, we love and live to serve and learn every day. That's kind of our vision and our motto or our mission. I’ve listened to YouTube videos all the time when I'm driving, whether it's a lecture or whatever it is, I read a ton of Audible books when I'm driving, I listened to them obviously. I read a lot of books. I also have offline things like providing coaching, and I love to mentor other people and I then I also love to be mentored. I had a great friend of mine mentioned to me that every person should have Paul and Timothy and their life and that's a scriptural reference for a mentor and apprentice in their life.
It's neat, but in general, I've thought about a mobile application that can actually capture all of these online learnings, but then allow me to enter the books I've read and other things like that. Does that exist yet? Can I go subscribe to it or should I go build it?
RUSTICI: Probably go build it. That was part of the really big vision of xAPI at the very beginning, that was now possible to build that app and allow you to track those things. A couple of companies started down that path, but they quickly found that most of the world isn't as motivated and disciplined as you are and that the motivation for capturing and tracking that just kind of wasn't there. You do have that as a feature of some tools. Like, Degreed I know has some of that capability built into their tool as well.
So, I'd love to see that happen. I do believe that what you've learned is going to become much more valuable than your resume in the future. Especially as an employer, I really don't care about what you've done because I know what I'm going to ask you to do is going to be different in three years. What I really care about is your ability to learn. If we can start quantifying that, tracking that and demonstrating that, I think there's a huge potential to shift the way we deal with employment and skills and market ourselves to the job market.
PARNELL: I love that. That's a great perspective. I totally agree first of all, but additionally, there seems like there are some applications out there that are getting close, but it's really hard to integrate.
A lot of those different systems are massive applications like YouTube or iTunes or Audible Pocket does a decent job. You can almost add anything on a phone, I think on your iPhone to Pocket. Then, real time monitoring of usage of different things is going to be a massive integration, a mission.
I love it. I think you're right. I think the future resume is a series of what you've learned in the and LinkedIn is actually getting close to that with the skills that you can actually label yourself with and people can star those skills that you have. Again, it is a little different than proving it – the actual titles and the videos.
Mike, I really appreciate you joining us today. Check out Mike at Watershed LRS. I found him on LinkedIn pretty easily. Mike Rustici, he's got quite a name in the industry having essentially helped the government build the SCORM standard in 2000. We just really appreciate you coming on the show today, buddy.
RUSTICI: A very much thanks for having me. I appreciate the opportunity.
PARNELL: Okay, thank you.
HENRY: Thanks for tuning into LITES, it’s Leadership in Industrial Technology, Education, and Safety. See more at LITES.org. If you enjoyed the show, please remember to rate, review, and subscribe. LITES is a production of Industrial Training International. Our guest today was Mike Rustici with Wastershed LRS. Our Producer is Michael Monatine.
I’m your host Mandy Henry, and we’ll see you next time.
