Unveiling AI's Transformative Impact with Tom Marsh: A Microsoft Product Management Maestro's Journey Through Enhanced Business Applications and Personal Passions

Mark Smith: 0:32
Welcome to the Power 365 show. We're an interview staff at Microsoft across the Power Platform and Dynamics 365 Technology Stack. I hope you'll find this podcast educational and inspire you to do more with this great technology. Now let's get on with the show. In this episode, we'll be focusing on large language models and their role in Microsoft business applications. Today's guest is from Redmond Washington in the United States. He works at Microsoft as a principal group product manager. He's technical leader with over 25 years of experience in product management and engineering roles. You can find links to his bio, socials, etc. In the show notes for this episode. Welcome to the show. Tom Hi Mark, thanks for having me. Was the introduction correct?

Tom Marsh: 1:19
Yeah, it sounded right.

Mark Smith: 1:20
Good, excellent. I'm excited to have you on because, at the moment, ai is my favorite topic. In fact, somebody pointed out that since 2018, it's been my favorite topic because that's when I really first started blogging a lot around AI and I was excited about where it's going. Then I just feel that Microsoft actually I'll tell you why I started getting into it then I felt that Microsoft was the only company out there doing what I call practical AI. Gpt has been in the power product for the last couple of years. It's not new, but, of course, since October 30th, when Sam Altman did what he did with OpenAI and everybody's got excited about large language models, the world has fundamentally changed. We're coming up 12 months, right, yeah, this month makes 12 months. Oh my gosh, what a whirlwind it's been. Before we unpack your role, what you do, etc. Tell us a bit about food, family and fun. What do they mean to you?

Tom Marsh: 2:16
Well, I live in Redmond, as you said. I have a wife and two elementary school kids, so most of my life definitely revolves around them. All my free time playing games, going to events, watching movies, taking them to classes, all that kind of stuff. So, yeah, that's fun. I especially like Minecraft. We get lost in hours playing that. Food-wise, I'm not a great cook. I do enjoy cooking, but my family has to put up with whatever I come up with. I guess a lot of omelettes, a lot of pancakes and things like that. But, yeah, have a lot of fun with that.

Mark Smith: 2:49
Very cool Minecraft. Do you have your own server set up in a jour?

Tom Marsh: 2:53
No, I just have the regular bedrock edition and just play around in there.

Mark Smith: 2:58
Okay, so my son is now 18 years old and I've lived around the world as in. He was from a former marriage, but one of the ways we stayed connected is I set up a Minecraft server in a jour and then we would play online no matter where I was in the world. Very cool as he grew up, and it was a really good way because you could have conversations etc. While you were playing, but you were playing the game, and so it was such a good tool, I reckon, to actually have good communication time.

Tom Marsh: 3:30
Yeah, I usually get my kids to lie on the floor in the room, so I'll be on my device and they're on theirs and we'll be sad in real time. But yeah, if they weren't local, then that would be a great way to do it.

Mark Smith: 3:40
Yeah, I love it, and being able to set that all up in a jour was just epic. But yeah, probably haven't played for a couple of years. Now that he's got older and about to start studying law, his focus has changed a bit. Very cool. Tell me about how you got into Microsoft. What's that journey been and bring us right up to what your role is today and what you're doing.

Tom Marsh: 4:00
Yeah, absolutely. So I joined straight out of school. Actually, I first joined as an intern because my roommate across the hall from me had had an internship here and just raved about the experience that they had had and sounded like a lot of fun. So, all right, I'll go check that out. I also fell in love and came back for a second internship, actually, and then came back full time. So it's almost the only real job that I've ever had. I moved around in the company several times, but it always has that same bedrock culture and approach, so it feels different jobs, so it's one job, family, if you will. Yeah, so you were there from Gates, he was. Yeah, so this was my first internship was in 96, and then I started full time in 1998. Amazing, yeah, and so the theme throughout for me, most of my positions have involved insights of one type or another. So I started as a developer working on an online learning tool, so trying to bring students' insights back when online learning was not so common as it is today, doing some of these kind of distance universities and corporate training, things like that. Then I spent a number of years doing insights for developers in Visual Studio, doing unit test generation and ethics, cop and pre-fast, basically helping you find issues in your software and correct them so that we can all enjoy more reliable software. And then I spent about a decade in Bing and Bing ads, most of that time working on the knowledge graph. There I used to joke that we were trying to learn everything there was to know about everything. So no matter what question you asked, we'd be able to answer Kind of before ChatGPT came along and gave you a natural language way of doing it. We were a little bit more structured, but that was a really fun time. And then, about two and a half years ago or so, I came over to business applications and thought there's a real opportunity to help people get more insights, be more efficient in their work. We all spend so much of our time working and so if we can have some impact on that, I felt it would be a really important thing to do, and I love it. I feel like every day I'm getting to hopefully get a little bit less toil in people's day-to-day life and help them get more satisfaction, get more enjoyment, get more done in their work. So that's what I do these days.

Mark Smith: 6:12
So how does that practically hit the technology that we might see or use in our day-to-day?

Tom Marsh: 6:19
Yeah, so our team is called Co-Pilot AI team, so we build the AI that is sitting between the foundation models and all of the applications, whether you're talking about Power Apps, power, automate and many others, or all the Dynamics ones and so the manifestation really is this idea of Co-Pilot, and Microsoft's vision for Co-Pilot is this idea of a real-time collaborator. That's not something that you just automate everything and go on vacation for a month. It's really about sitting there with you, embedded in your flow, understanding your context and helping you get done whatever task you have. And so that manifests different ways for us depending upon whether you're working on an email, working on a sales deal you know will help you write the email or whether you're working on building a Power App. Then it will help you either create the table behind that, give the right instructions to GPT to help build that table, or maybe arrange the controls on the screen of the app and edit those as well, and even let you add Co-Pilot to the app that you're building so that your users can also get a Co-Pilot.

Mark Smith: 7:21
Yeah, so you're saying that this is the intermediary between if I was using Co-Pilot in Power Automate or if I was using it in Power Apps. It's the working piece that applies across the Power a suite of tools.

Tom Marsh: 7:33
That's right. Yeah, it's not quite as unified as that just yet. We're getting there. It started out by a lot more pieces, right, okay, let's go and build one thing for Power Automate and now let's do something slightly different for Power Apps. But yes, over time those are becoming more unified and kind of a more common layer of abstraction that sits in between the foundation models and the apps themselves.

Mark Smith: 7:54
So does it have awareness of me in that context? And what I'm alluding to here is that, you know, the Office Graph kind of knows what I'm working on, what documents, teams et cetera, knows the calls that I'm having, potentially with customers, things like that. Is it feeding that into the awareness or is it only purely based on what you're prompting to do, or is it understanding me and my working behaviors and how I could be more efficient?

Tom Marsh: 8:23
Yeah, it's getting smarter and smarter over time. So the first versions were fairly unaware of context, right, you know, think static prompts that give you some pre-canned output. And then it got more advanced. We started doing things like fine tune model for Power Automate to create a flow right, where we trained that model specifically on many, many examples of flows and descriptions of flows. And so now it started to feel a little more contextual, because you're describing what you want and you're getting that flow out. But the reality is the underlying model is the same for you as it is for me. And then over time we're starting to add more and more context. So now, as you go in and you're editing a Power App, looking at what edits you are currently making, what is the app in its current state, and using that to feed into the prompt. We also use examples. Currently they're not specific examples per customer, but over time they will start to be more specific examples per customer and you can imagine that being at a tenant level or even at a user level, so that those then get fed into the prompt, is not that the underlying foundation model itself is changing to know you, but the business applications are getting to know you by injecting these examples and kind of teaching it and call it in context learning, so that it gets smarter at the time that you make the call.

Mark Smith: 9:46
Yeah, interesting. Tell you, the favorite feature of this for me and I tried it must have been back in March this year. I was on a live session with Charles Lamana and he announced of course, this is the MVP, session was a big, the co-pilots you know is coming, and that you could build your own app. And so, while he's live talking, I jumped into my machine and I built an app, a Power App, for managing my seed library. I plant lots of, I'm into gardening lots of seeds and I've never found a tool that does it well. And then, if I put my day job hat on, I mainly in pre-sales right, so I am taking a customer scenarios and wanting to show how they light up on Microsoft Technologies. What I loved about it was the ability to build sample data that was on point with what my situation was, and that alone that feature alone, for me and my role and what I've done for the last 20 years in pre-sales, was massive Exactly Massive game changer.

Tom Marsh: 10:51
And I think a lot of people have these stories where it's just that one really surprising, delightful experience that they have with these models. Right, they're still imperfect. Right, there's a lot of things they can't do, but I think we've all experienced that kind of magical example, not just once, but repeated magical examples that just make them a delight to work with.

Mark Smith: 11:10
Yeah, I like the metaphor that Mo X Microsofty, who's written a book on AI and whether we're going to have a utopian or dystopian future within the next 40 years, and I like his concept and it really grounds me is that where we are at with AI right now is really an infant child, almost just a toddler right. And like I see, last week Sam Altman has started to talk about GTP5, gpt5, right and some of the challenges and stuff around that and like we've had one year GPT 3.5 Turbo and then four and that's just in the first 12 months, right.

Tom Marsh: 11:47
So this baby is really starting to grow.

Mark Smith: 11:50
right Now, back to what I set off air to you about Orion Cunningham and his question to the audience yesterday around where will you be in three years time? Or what will you be doing? Or, and I go back to where we are in our journey of AI. That's become although you know, we know AI has been around since the 60s odd and maybe even before, but our concept of it now in the context of large language models and really it becoming highly practical, definitely changed the workflow of my day to day in so many areas. In fact. Sneak thing here about the podcast that's been running for six years and I'm not the best writer of titles for each of my podcast, right they after six years twice a week, it gets pretty generic. Since I have now AI read the entire transcript and come up with five potential titles for the show, I've seen over a third increase in traffic to the podcast because of the title rewrites.

Tom Marsh: 12:53
Yeah, and, can you imagine, would have been like to do that just a year or two ago right, you would have had to go and create a custom model, you would have had to go and get a bunch of examples of transcripts and a bunch of examples of labels. Like you'd never do it, it was just be way too much work. But that's the magic of these models is that they're finally big enough, general purpose enough and easy enough to deploy that you can use them all over the place, right, yeah, and that's the explosion that we're seeing right now and what makes it so exciting to me.

Mark Smith: 13:20
When you look from your position inside Microsoft, looking out, and you're seeing the data sets that you're analyzing and perhaps the industries. Are you seeing any kind of industry and market adopting this quicker than others? And particularly thinking what are the use cases that are coming to the surface? I love I think there was a stat yesterday that showed it's a 2x improvement on writing power automate, so creating a flow is two times as fast now. That's massive, right. If you imagine everybody could 2x their day from a productivity perspective. What are the things that you're seeing? What are the trends?

Tom Marsh: 13:56
Yeah, it's really across the board, I think. We see customers from all walks of life who are approaching this. I mean we even see some financial services companies that you might think, oh, that's highly regulated, maybe they might take a while to jump in, and they are, of course, more cautious, but they are also finding ways to bring value, let's say, to their own employees. So how do they deploy? Let's go and use Co-Pilot to create a bot that you used for employee facing work as a way to get started and still get a ton of value out of this. We see communications companies also looking at how to improve, kind of get more people coming and building automation, and how Co-Pilot makes it easier for them to do that, and so it's less time, it's more people being able to access these things. We see customer service. We see our own customer service group getting significant wins in terms of how long it takes to complete a case and, of course, you see, also customer satisfaction goes up there as well, both because of the accuracy but also because, hey, and who wants to wait longer for their case to be resolved? Just by getting it faster, the satisfaction also goes up. So it's really it's kind of all over the place that we see this taking off.

Mark Smith: 15:12
How do you your role, because, as somebody outside Microsoft, they feel like I'm drinking from a fire hose right this year, more than any other, you know, how are you maintaining the level of information, continuous learning, that you must be on because of the speed that things are going at?

Tom Marsh: 15:36
Yeah, it is definitely feels like a fire hose, right. I mean, I think we're very privileged to sit where we are because we have so many different sources of that information. Right, we're learning from our customers every day, talking about what they're trying to accomplish, talking about what they're seeing or hearing from other vendors. Obviously, we have the news itself constantly bringing us all kinds of information. We have partnerships, and so we hear from our partners on a regular basis what they're thinking about, where they're going and have conversations with them about that. We have our own research organization. We have our own product teams. Obviously, microsoft's a very large company, so there's a lot of different people, and Co-Pilot, I think, is really the only example in the last 25 years that I can remember where it felt like a whole company thing, and Microsoft is probably almost an order of magnitude larger than when I joined, and so the whole company means something different now. But almost everybody I run into, almost every team, has some aspect of Co-Pilot that they're working on, and so there's a lot of this kind of cross group exchange of ideas and it feels very collegiate. It's all these brown bags and internal events and things like that on top of all of the external ones, and so you just have to pick and choose. There's plenty of information out there. It's not like, oh, I can't find it, oh, I've never heard of what's going on. It's more a question of how do you prioritize and how do you choose where.

Mark Smith: 17:03
To go a little bit deeper that's an interesting concept of you talking about this very collegiate type of environment, because I've worked with Microsoft for 30 years now in the ecosystem and of course it's not always particularly between product groups had a love on each other type relationship and it's interesting how the spirit of technology doesn't matter with the M365, whether you're in Azure, whether you're in, I assume, even the gaming and the product divisions, r&d, et cetera. It really is permeating every part of the business and I assume that the collective knowledge, the cross pollination, right, and so they'll be looking at it from a very specific area and you see that you're like you know what we could use that Are you seeing a lot of that? Like I hadn't thought about it like that and therefore we need to bring this to our group.

Tom Marsh: 17:49
Yeah, absolutely, we see that all the time right Learning from what Office is doing. In our early days we took a lot of inspiration. Early days, a year or two ago, we took a lot of inspiration from what GitHub Copilot had done, because they were sort of the first on the scene with this and I still am inspired by a lot of the work that they did in terms of how they determined what aspects of Copilot made the biggest difference in how much people loved the product. They did some really interesting things, comparing qualitative studies or surveys and the quantitative behaviors that those people were doing, so that they could figure out one that matched the other, and so we took inspiration from that. But there's a very concerted effort to create these forums across the company, at the very senior levels, but also through things like internal machine learning conferences and other more Copilot specific forums, where you could probably spend a good part of your week attending these things if you had the time. Yes, we definitely do get a lot of inspiration out of those forums.

Mark Smith: 18:50
And what you're doing is hardware slowing you down at all.

Tom Marsh: 18:53
Not too much. I would say there's two aspects of that. There's one is what's the throughput, and the second one is can you get the hardware? Yeah, so far I think our supply chain folks have been challenged for sure. But I think that our leadership saw the opportunity early enough that there's been enough hardware to go around and at least in my team, where I am, we've been fortunate enough to have what we need. In terms of speed, it's hard to get too caught up in it, for two reasons. One is the models just keep on getting faster and faster, and not by 10% or 20%, but by 2% and 3x. And the other thing is that when you're saving somebody minutes or possibly hours of time, if the response is a little bit slower, it's not such a big deal. I find myself sometimes going and asking Copilot something, and then I'll go switch. I'm not the most patient person in the world, so I'll go switch and maybe start reading an email or something and I'll come back, and it's a perfectly reasonable interaction model to me. If you'd asked me whether a multi-second delay was acceptable when I was working and being out, I told you you're crazy. Nobody's going to come to our site anymore. Yes, yes, yes, but here it's a very different thing, because even with those kind of delays, the time it would have taken me to do the equivalent without Copilot was so long that I'm still saving a huge amount of time and I keep on coming back.

Mark Smith: 20:17
Yeah, amazing what challenges, and I want to put the ethics and responsible AI to ones like I think a lot of people have indexed on that. But more from a whether it be speed, whether it be adoption, whether it be technical blockers, what are some of the things that you know are some of the challenges that you're going to need to solution for and I'm alluding slightly to Sam Altman he was talking about there needs to be changes in the hardware and stuff. Do five needs to do. There's some scientific research. It's not there yet. Are you thinking about these things in the context of the business applications ecosystem that are like guys, like when we look at backlogs and things? This needs to be a priority. This will become a roadblocker if we do not solution for it.

Tom Marsh: 21:11
Yeah, I think a lot of the energy is actually going into what I might call a block in tackling. It's about how do you get rid of the grounding errors, for example, yes, the default models have still a significant issue with poorly grounded responses, which is not okay. If you're creating a unparsable flow or unparsable app or you're giving somebody just the wrong answer and they're relying on that in some financial transaction or something. A lot of energy goes into how do you get the right retrieval, augmented generation or other techniques to get the right examples, the right data and then validate into the model and then validate that the output actually matches those examples so that these can be used in a business context. If you just go over to a regular model without that grounding, you're going to have some pretty serious issues. That's a big part of it that we look into. Then, even once you fix that, there's still another layer of I don't know how to describe it, but there's quality issues. Right, the response is not quite right. Additional work goes into things like how do we rewrite the queries to give us better results or how do we iteratively fix the results that come out and say, okay, this is wrong, please go and fix this, instruct the model to go and fix this aspect or that aspect and kind of iterate until we see a better result. We do a lot of experimenting with different ways of prompting. There's a great doc that one of our data scientists wrote. That's basically. It's like the prompting equivalent of coding guidelines People talk about hey, english is the new coding language. Well, great, we need some coding guidelines for it. How do you word things? What order do you put the instructions? What kinds of additional guardrail instructions do you add onto it? Is it big, important piece as well? As we try to push for this? Accuracy. Accuracy is really the number one thing that we're after. It's like accuracy, and not just on simple things, but being able to handle very complex tasks, where you start to give a vague instruction for what you'd like your application to do or something like that.

Mark Smith: 23:19
I just started playing recently with Dali and the reason is it's now in the GPT-4. And I noticed that after it created the image, I then would have an information and actually had fully rewritten my prompt. Yep. I was like, if we dial back nine months ago, everyone was like, hey, you got to become a prompt engineer. I just feel that AI will be the best prompt engineer itself. In other words, one of the things I instruct my AI to do is always ask me qualifying questions and never just reply qualify me, qualify me until you're satisfied that you're understanding what I'm asking. It's interesting what you say. There is really understanding those, then queries behind and how they could be written better and perform better. I'm probably asking you to speculate just from an observational perspective, not from your professional perspective. But low code, no code. If I look forward, when I saw that Ryan Cunningham had that up there, I truly feel that the idea of low code would be gone within five years and it will be no code from the context of how we look at the Power Platform, because I assume we're going to get to the point that I'll be able to give a descriptive instruction, get it qualified and then a very sophisticated app could be built Absolutely, or would it be an app? It might suggest that this should be a chat bot or the interfaces, etc. There would be a certain choice based on the targeting personas and things like that. Do you think that, and what you've seen in 12 months and you extrapolate that out over the next five years three to five do you see that we almost can't predict where we will be there? Do you think it'll be that quantum leap different than where we are now?

Tom Marsh: 25:15
I do think it's very hard to predict where we're going to be in the future, but I also agree that with what you said about low-code going away at least I mostly agree with it I think that it is very likely that in that time frame, or even sooner, we'll be able to build fairly complex experiences using just natural language and probably other modalities too right, speech, gesture, images, videos, things like this to communicate, just as I would communicate to you if you were some magician building something for me.

Mark Smith: 25:48
Yeah.

Tom Marsh: 25:49
I do think that will work, but I think that the other piece that needs to be retained is there needs to be some way for the machine to tell me what it is doing, or show me what it is doing and I don't know whether you call that low-code or not, because I don't think that necessarily showing me a flow diagram with a bunch of technical jargon on it is how I want to see that. But nor do I want to listen to one hour lecture that is painstakingly describing to me. Okay, this is an app that has five buttons. The first button is called like no, just show me, give me a walkthrough, give me like, what a YouTube video on this app would show me about how this app works. I think that's maybe the future so-called low-code experience is that you describe how you want and you get a multimodal description back of what it has actually created for you and probably also even some ideas on how you might want to make it better. But truly, it's collaborating right, because there's this idea of sparking creativity that is a really important part of co-pilot. It's not supposed to be this, just this kind of instruction follow-over, where you tell it exactly what you want, it does just that thing, and then it kind of like dumbly, sits there right, it's supposed to anticipate and spark oh, here's more that you might want to do and give you that idea. So you get excited and like, oh yes, that would be great, I want to do that and I want to do this other thing, and you come up with something much better than you would have on your own.

Mark Smith: 27:11
Brad Smith a couple of years ago, made a 2019 release to book and, of course, long before where we are now, and he said some people liken data to oil. Right, and he was like it's wrong, it's more like air. Data is more the air we breathe. And what's your thinking around the importance of the data estate that you're operating over? You talked about grounding before. What's the importance of one access to the data that needs to inform decisions to the robustness of that? Is there enough? Is it the most recent? Is it? What's your thoughts on the importance of data in an AI future, for the way any kind of business should be considering how they adopt AI.

Tom Marsh: 28:04
I mean, it's absolutely essential. Without accurate and complete and up-to-date business data, the AI can only guess. Now, it can guess pretty well, maybe it can guess better than a human would, but it's still guessing. It's not gonna guess exactly. How many widgets do you have on hand? How are you going to know that if you don't just go and look it up, right, and so that information being there is really important. I think the good news, though, is that AI not only depends on that data, but it can also help you create that data Right. So it can help you create like. I don't know if you played around with the summary and teams, but now almost every meeting I go to has good meeting notes. When was that ever true? Like, I mean, there was always that one person in my team who was very diligent and would take good notes, and all the other meetings were just junk right, and now Co-Pilot is actually pretty darn good at it, and so every meeting has notes. That is a form of data that is very important to capture, because it can also start pulling out action items, but you look at it in that business context. You know we're pulling out contacts and price requests and things like that out of emails and actually get your CRM into good shape right and allow you to. You know we can normalize the data much better because it's getting AI created. It'll be much more comprehensive because you're not having to type it all in. It'll be much more consistent because it does it for you, and so there's a virtuous cycle that is starting to be created where the AI is creating better data, consuming that data and getting better at creating that data, and I think we'll see that flywheel continue to its kind of natural conclusion.

Mark Smith: 29:38
Yeah. I like it. In closing, tom, is there any advice I suppose that you could give to kind of three audiences that listen to this show that? My three audiences are people using Microsoft business applications, that's the Power Platform as well as the Dynamics 365 Suite. They are then consultants that generally working for partners implementing these technologies as part of their career. And then the third group, often Microsoft sellers in the field that want to understand more. You know they can do it on their commute and things like that. With that audience in mind, what kind of recommendations would you give them around how they can keep themselves or what should they be focusing on when it comes to that fire hose? You know, because they want to stay current, they want to be, they see a future where they're going to be fully fill in with AI and it's going to be part of their life and they want to adopt it. Yeah, what advice.

Tom Marsh: 30:36
Yeah, I mean, I don't think there's any substitute for just using it right. So try and figure out how to get your hands on this and make time in your day to go and use it, because it'll help you get past the hype and understand okay great. Yes, it does have some flaws here, but it also is really amazing in these cases and it'll allow you to make better decisions when you're talking to your customers or your colleagues and say, hey, I think you should use it in these 10 ways and maybe not in this one or two other ways that are not quite ready yet. Because it is, the space is moving so fast that if you're just trying to kind of like stay on the sidelines and watch it go by, you'll miss it. You need to jump in, get in the stream, play with it. Especially in Power Platform is really easy to get started, but also in Dynamics as well, if you can get yourself an instance and try it out and watch that. And then, besides that, we also have, you know we're doing more and more content creation for Microsoft partners. A lot of the folks you just listed would have access to that content. We have this site akams slash BA, copilot, as in business applications copilot. So many demos, presentations, documentation. We're constantly putting information up there, so I would watch that as well, but I think there's really no substitute for just trying it, using it day to day yourself.

Mark Smith: 31:55
Hey thanks for listening. I'm your host business application MVP, Mark Smith, otherwise known as the NZ365 guy. If there's a guest you'd like to see on the show from Microsoft, please message me on LinkedIn. If you want to be a supporter of the show, please check out buymeacoffeecom. How will you create on the Power Platform today? Ciao?