Expert Panel: Learning Transfer, Chatbots, and Measurement

Learning Transfer Webinar

On Nov 20th, we were pleased to host Trish Uhl, Emma Weber and Paul Bills for a panel discussion on chatbots, Coach M, behavior change, learning transfer, and measurement. This great discussion, hosted my Mobile Coach CEO, Vince Han, provides valuable insights into one of many effective uses of chatbot technology for learning and training purposes.

About the participants:

Emma Weber – Linkedin | Twitter

A recognized authority on the transfer of learning, Emma has been a guest speaker at conferences in Australia, New Zealand, Malaysia and the USA. She shares her passion and expertise through her writing, and her first book, Turning Learning into Action: A Proven Methodology for Effective Transfer of Learning, was published in 2014 by Kogan Page. Her second book, Making Change Work: How to Create Behavioral Change in Organizations to Drive Impact and ROI, co-authored with Jack and Patti Phillips of the ROI Institute, was published in 2016 by Kogan Page.

Trish Uhl – LinkedIn | Twitter

Trish Uhl, creator of the Learning Systems Engineering Framework™ and founder of the Talent & Learning Analytics Leadership Forum, works globally with learning leaders and business executives on transforming their organizations into Digital Workplaces and developing their people’s digital fluency and dexterity. Trish’s unique skillset combines data science, learning science, advanced analytics, artificial intelligence (AI), predictive modeling, ambient intelligence, voice UX, blockchain and emerging tech to promote positive people impact and deliver organizational performance.

Paul Bills – LinkedIn | Twitter

Paul Bills is Director of Chatbot Design at Mobile Coach. He has a BA in English from BYU and a Master’s of Entertainment Arts and Engineering in Video Game Production from the University of Utah. Paul has spent time as both a poet and a videogame designer. Now, he uses skills from both to create chatbots.

Vince Han – LinkedIn | Twitter

Vince Han is the founder and CEO of Mobile Coach and a frequent speaker at conferences such as Training Conference, DevLearn, Learning Solutions, Masie’s Learning Conference, ATD ICE, ATD Techknowledge and others. He holds an MBA from the MIT Sloan School of Management. Vince is an industry thought-leader for learning and learning technology with an emphasis on artificial intelligence and chatbot technology. Vince has founded several successful technology companies and resides in Utah.

Webinar chat discussion transcripts with time code and link:

00:09:41
Trish Uhl: LOL

00:10:44
Trish Uhl: Curious – what kinds of chatbot Web site experiences? Banking? Retail? Travel?

00:12:07
Vincent Han: Please ask questions here or in the Q&A space!

00:12:16
Trish Uhl: me too @Craig

00:13:38
Trish Uhl: nice @Molly that’s a broad range of experiences

00:13:53
Trish Uhl: nice @Suzanne

00:14:37
Trish Uhl: have you had a good experience conversing with a chatbot?

00:14:59
Trish Uhl: text-based @Stephen

00:15:12
Trish Uhl: this iteration 🙂

00:15:29
Trish Uhl: My pleasure 🙂

00:15:47
Sharna: Wonderful to participate in conversations which appreciate that learning is about behavioural change – not consumption.

00:15:58
Colin: Re: having a good experience – I have to admit that my experiences have been pretty poor. The chatbots I’ve used on public sites especially have been very shallow. From a UI perspective, the affordances are unclear – it’s hard to predict what questions the bot is equipped to answer. I find it much more efficient to scroll through as list of FAQ questions.

00:18:55
Emma Weber: At a basic level Theresa we measure against three objectives that an individual sets

00:19:30
Emma Weber: Thanks Jen

00:25:08
Suzanne: Re: my experience has also been pretty poor. I’ve generally used them for FAQ where I eventually end up needing to speak with a live person. However, I did have a good experience on a healthcare site where I was directed to order the correct lab work and it flowed right into a successful online order.

00:25:13
Trish Uhl: I can see that too @Colin – with a simple FAQ – but I’ve seen some cool use cases of chatbot used to pull answers from multiple disparate data sources – then you’ve got a unified interface to curate relevant content Have you had that experience?

00:25:35
Vincent Han: No slide but I’ll ask Paul to summarize here in the chat

00:27:10
Suzanne: Is anyone aware chatbots being used in medical education or physician continuing professional development?

00:27:19
Colin: @Trish – I’ve seen attempts at that experience, but again the discoverability problem is a serious barrier.

00:27:50
Paul Bills: 3 questions: 1. Can the problem be solved? 2. Can a conversation solve that problem? 3. Can that conversation be automated?

00:27:56
Trish Uhl: agreed @Sharna! #behaviourchange

00:30:11
Trish Uhl: It’s a good question @Stephen What Emma has been able to show with Coach M (specifically) is that engagement and retention are comparable to the experience she and her team have had with one-on-one human conversations

00:30:59
Trish Uhl: Then tracking things like ongoing engagement over a period of time — how long do people stay in and participate? Again, specific to Coach M, it’s cohort-based – so people are meant to interact with over specific period of time

00:31:46
Trish Uhl: but other analyses can include sentiment and NPS

00:32:21
Trish Uhl: you can analyze chatbot conversational data for sentiment, as one example

00:32:45
Trish Uhl: we can also use qualitative conversational data for predictive modeling – to predict things like cohort performance

00:33:13
David: Emma would you say Coach M is better at being a) personalised to the user or b) expert at learning transfer process or both?

00:34:18
Suzanne: Are responses to reflective questions stored/tracked somewhere? Or can they be?

00:34:24
Brent: COHORTS FTW!!!

00:38:00
Trish Uhl: @Colin sentiment analysis can be used in pairs (as one example) so when people are making progress or not, what are their language patterns – and what sentiment do they seem to have?

00:39:18
Trish Uhl: So — we can actually use those language patterns to optimize both training content as well as how we communicate / market that training by using the actual language our people use when X is happening – know what I mean?

00:40:26
Trish Uhl: The language patterns, @Ian?

00:40:36
Emma Weber: Piotr – thoughts, feelings, values, beliefs, fears and needs 🙂 under the iceberg stuff…

00:45:38
Trish Uhl: @Colin and @Ian – cultural nuance is one area, as example

00:45:56
Trish Uhl: Then you’ve got language nuance too when we get into translations

00:47:07
Trish Uhl: So – we do due diligence in crafting conversational experiences, but we know words matter – and we can help optimize the language based on analyses of the patterns used – just like how marketers do with consumers 🙂

00:47:39
Trish Uhl: @Emma has done some tweaks on Coach M specific to language — let’s see if she has an example off the top of her head

00:49:35
Trish Uhl: also interesting how people interact with Coach M in particular – they thank ‘her’ – and wish ‘her’ Happy Christmas

00:50:52
Trish Uhl: those reactions to the bot are signals in the data too

00:51:39
Trish Uhl: @Ian you asked about the scripting — something like Coach M has 127 different potential conversational combinations – so no two conversations are alike

00:57:48
Trish Uhl: Gotcha @Ian! I’m familiar with Eliza – and Woebot

00:58:03
Trish Uhl: I know Woebot is more CBT — and Coach M doesn’t go there

00:58:40
Trish Uhl: It’s literally designed to prompt self reflection and through questioning techniques all within scope of individual’s Action Plans

00:59:15
Trish Uhl: Speaking of chats …. what takeaways are you getting from today’s session?

00:59:26
Trish Uhl: >> What are your key takeaways?

01:03:00
Trish Uhl: Thank you all!