Revolutionizing Secure and Scalable Data Ecosystems

Mark Smith: 0:01
Welcome to the Ecosystem Show. We're thrilled to have you with us here. We challenge traditional mindsets and explore innovative approaches to maximizing the value of your software estate. We don't expect you to agree with everything. Challenge us, share your thoughts and let's grow together. Now let's dive in. It's showtime. You know we're all in remote locations today, but the show must go on. As they say Commitment to the cause.

Mark Smith: 0:29
Andrew, a couple of weeks ago, you released I think it was at the Brussels conference the white paper around data, which, for anyone that hasn't seen it, I just you know that in our industry industry. You've got to check out this white paper because it articulates so clearly the five layers of integration, or the five ways from, you know, medium complexity to well complexity is probably not the correct term for it. It goes from simple integration to advance. Therefore, you get more value the further along you go you create a data landscape for your organization. So, andrew, can you tell us a bit about your thinking around it? How did it come about? What was your? And I assume, anna, you've had a lot of input to it as well in that white paper you've had a lot of input to it as well in that white paper.

Andrew Welch : 1:29
Yeah, so the white paper is called Power Platform in a Modern Data Platform Architecture.

Andrew Welch : 1:31
And it's actually interesting. I went back and I looked and part of the reason that we titled it this way was that in early 2021, I published an essay, so it wasn't quite white paper length, and it was called Power Platform in a Modern Data Platform Architecture. And to this day, I mean we're, you know, we're years later and that is by far the most popular visited search term thing that I've ever published on my blog. So you know, I wanted to take the topic up again and I think that one of the a couple of the big challenges that organizations have when it comes to really scaling and using power platform across a big enterprise organization is always this challenge of well, apps are only so useful if you're not able to use them with your organization's data right. So, okay, where do we proceed from there? And this was something there were sort of two parallel tracks.

Andrew Welch : 2:34
Anna and I have been working on this topic for about a year now, maybe a little longer, and we've been presenting at conferences.

Andrew Welch : 2:42
We presented the topic at Nordic Summit in Copenhagen last year, at Dynamics Minds, at ColorCloud, at European Power Platform Conference in Brussels. So we've had a lot of opportunity to talk about this with folks out in the world and in the field who are doing this work and to really refine the model. So that's what the white paper does. The white paper goes into these five patterns of integration between Power Platform and the rest of your Azure data platform, with a particular emphasis on fabric, and it lays out these patterns from the more straightforward kind of the simple point to point, using data connectors and virtual tables, which I think probably most people are familiar with, all the way up to a quite sophisticated how do you actually build Power Platform into the I won't say fabric, I'll say into the fibers of your data platform? And this has been a topic that has really resonated just if you look at how many hundreds of people are overflowing out of the room when I present it. So I'm happy it's written down now in white paper form.

Mark Smith: 3:52
And this goes further, though, than just the Microsoft ecosystem. Right, it could be your HR system. It could be Workday, for example. It could be SAP. You know data estate. It's around bringing all the data sets together and ultimately making them available in a format that's consumable, that's secure, that's repeatable, that's scalable as your organization develops new ideas, products, solutions, services, internally for staff to use. Is that how you see it? Bringing all those data sets together?

Andrew Welch : 4:35
Yeah, and I just looked this up because I wanted to quote this. The quote from the introduction is for all the talk about Power Platform as a low-code tool and then I promise I would not use that phrase again because we're trying to move away from low-code and I didn't in the rest of the paper but for all the attention given to how supposedly easily it allows non-technical users to create simple apps. Power Platform's greatest value lies not in the app, but in the data the app collects or serves back to its users. Simply put, power Platform isn't an app phenomenon, it's a data phenomenon.

Andrew Welch : 5:10
And you know, as I spend more and more of my workday nowadays helping organizations build big global data platforms, what I am finding is that it's very difficult to build a data platform today that reaches its full potential without also building power platform into that data platform.

Andrew Welch : 5:35
That is, application data that's collected inside of Dataverse is just one hop or one sort of technical push from Dataverse into one lake or into a traditional Azure data lake. You know Gen 2 storage and because of that, power Platform really solves, I think, the last mile problem in your data platform, which is either. Well, maybe the first mile, which is, how do you get data in the platform and then the last mile is how do you get data out of the platform and provide it to users in a way that is actually valuable and useful? So power platform and data platform, I think, are inextricably linked at this point, and that's really what you're doing is you're trying to bring these data sets together in a way that allow you to extract value from them, and you can do it without Power Platform, but it's way more expensive and way more time consuming and, I think, way more fragile or rickety if you're trying to rig up solutions that don't involve Power Platform because you've not adopted it fully.

Mark Smith: 6:46
Yeah, and so when you're thinking, I just thought I was thinking of the size of organization that would be adopting this. We're not really talking about the SMB end of the town, right, you know they're going to be fine, probably just with Power Platform platform. We're talking about more the SMC enterprise space, right, where one they would have the affordability potential to scale and deal with large data landscapes, which, in my experience, I don't see so much of the SMB space.

Ana Welch : 7:18
I believe that we're dealing with the SMB as well.

Mark Smith: 7:21
Wow, okay, tell me more.

Ana Welch : 7:23
That's what I sort of wanted to add there. I believe that the best, I believe that the most important feature, you know, of the white paper and of the information shared, is that flexibility of changing your you know, changing your thinking. And the reality is that many, many partners out there, even though they're potentially dealing with enterprise, with big enterprise and like SMCs, they don't always get that deal, that big deal that would allow them to, you know, go forward with that fabric integration or to set up a data landing zone. But the beauty of it is that the five principles of embedding the Power Platform into a modern data architecture is that it's a practical tool to help you decide when is your simple integration technique not appropriate anymore. To give you an example, let's just say that we are connecting an app yeah, we are there building an app. It's going to be a model-driven app. And let's say that we're connecting it to an Access database.

Ana Welch : 8:44
We've all been there Like, of course, you can say, no, I'm not going to deal with such a small project or I'm never going to. You know, I'm looking into big enterprise solutions. But the reality is that whenever we go to conferences and talk to people, a lot of them do have this case scenario as well. So what do they do? Well, using the techniques presented in the white paper, they may say, okay, so I'm going to use a point-to-point connector to begin with, because I only have, because I only need the name of a contact, and that's what I'm going to do.

Ana Welch : 9:22
But, as it is, you might find out that your app is, let's just say, a clocking in and out Canvas app. Yeah, like super simple. You're getting your data from Access Database initially, and then you're like OK, I should move it into Dataverse, because actually I need it for my HR files. Right, I should move it into Dataverse, because actually I need it for my HR files, right? So at that point in time, you're like, oh, maybe I should go ahead and do a bit of data consolidation and you're going to you know your second principle of data integration, and then, step by step, you can actually evolve into creating a full-on data distribution neighborhood. Why not? Even though we are talking about an SMB here who purely just want to adapt, if we're doing our job correctly, there's no reason why these principles couldn't be applied for big, or indeed small, projects.

Andrew Welch : 10:23
I think that Anna's right there and I actually, as she was talking.

Andrew Welch : 10:32
There are one of the beauties of this technology as it sits today and as it has evolved, is that and when I say this technology right, I'm thinking specifically of Power Platform and Fabric working side by side One of the beauties of it is that power platform and fabric make the app plus data story, on the one hand, more easily accessible to smaller or less well resourced organizations that maybe don't have the expertise or don't have the the you know, aren't in the financial situation to invest in a big data platform, but, at the same time, it makes that same thing highly, highly scalable to organizations that do. As I think about it, actually, you know, we have these five patterns of integration between Power Platform and Azure Data Services, point to point data consolidation, master data node, data landing zone and data distribution. I actually think that the two that are most useful in the SMB space are number two and number five. Number two data consolidation, because SMBs have an opportunity to really consolidate all of the random spreadsheets and access databases and trackers right, that are floating around every business.

Ana Welch : 11:49
But they still don't have a huge volume of data, so that allows them to like, do it in two or three sprints. You know it's like tangible, exactly.

Andrew Welch : 11:59
Exactly so. I think that an SMB can really quickly get a lot of their data consolidated inside of Dataverse and then start building apps on top of that for however many people work there five people, 50 people, yeah. Flip side from a data distribution perspective, an SMB can then take that data from Dataverse, shortcut it into OneLake right, secure it in OneLake and start to perform data distribution type activities, whether it's analytical workloads, reporting, whether it's you know if you've got the technical capability there to start augmenting a large language model, or to use your own organizational data in the context of a copilot or what have you. The ability to do that now is now more accessible than it has ever been before. That now is now more accessible than it has ever been before, even to organizations that do not have a team of data scientists or database administrators sitting, you know, within their walls. So, yeah, I think it's widely applicable across both enterprise and SMB.

Ana Welch : 13:18
And then you take that scenario where you have the concept of a mini startup within a big enterprise organization. How many times do those startups actually fail? Because, after they have succeeded, they cannot be integrated within the greater good, they cannot follow the same processes, they cannot have the same data structures, they cannot afford their leaders the same results. So this is why I believe that the most important thing that can be immediately lifted out of that white paper is the principle of, or is the ability of, knowing. Ok, when should I make my data integration technique more complex? When is that time? What sort of tools should I use? How much data have I got, et cetera, et cetera. What knowledge do I have?

Mark Smith: 14:44
Yeah. So I tell you, what I took away from what you guys just said is that, and particularly for that SMB who would traditionally probably start with hey, we've got SharePoint and our M365, and let's put our data in there and let's build our apps over that, what I'm hearing you clearly say is listen, so start with Dataverse, get everything and make a choice so that you're only going to use Dataverse because that will give you access to all the other four layers up from the. You know from that level. Do well, one technical debt you're taking on. Two, you're going to have to change your mind later on, no matter what. So therefore, you're going to create a higher cost where making that choice from the get-go to start with Dataverse is going to open up your ability to transition up the stack as your organization matures. Did I get that right?

Ana Welch : 15:53
Yeah, I believe you've narrowed it down really well.

Andrew Welch : 15:57
Well, and I also think and it's funny given how little time I've spent in the SMB, smc space almost all of my work historically has been enterprise. At least all of my customer or client-facing work has been enterprise. At least all of my customer or client-facing work has been enterprise. But listen, as I think about this, actually one of the big objections historically to using Dataverse, or one of the big obstacles that organizations have had to overcome, has always been a cost obstacle. Right, and in a weird way I think that big enterprise, if you're a Fortune 100 organization some of the largest global organizations they can afford wall-to-wall premium power platform licensing.

Andrew Welch : 16:41
Conversely, if you're an SMB, think about what that really means Like, even at sticker price, right?

Andrew Welch : 16:49
So with no Microsoft discounts at all, your annual cost of getting one person a premium Power Platform license is $240 US, right, if you have 10 employees, you're talking about $2,400 for a whole year of all-you-can-eat Power Platform, which you know.

Andrew Welch : 17:10
I would argue that the productivity gains that an SMB you know with, say, one to 10 people, or even if you extrapolate that out to you, start to double that or triple that right, the productivity gains that an SMB can derive from premium power platform using these approaches and using Dataverse far outstrips the $240 a year US dollars per year that you're going to spend on the licensing. So I actually think that, in a weird way, the organizations that have the toughest time overcoming the cost hurdle when it comes to premium power platform are the ones we will call them the SMCs right, they're the mid-market organizations that lack the financial resources of the big guys but have so many more folks that need the license than the teeny, tiny organizations I mean. For us at Cloud Lighthouse and we're business partners. So you guys know this, this we've got wall to wall to wall premium power platform and we stick everything in Dataverse.

Mark Smith: 18:18
It's beyond valuable it's interesting that mindset, because just in the last couple of weeks I've helped a non-for-profit get set up on M365, to start with right and then, of course, with um power apps tailored for them, which somebody in the community, a guy in wa, australia has gone and built, uh, out of the goodness of his heart for this non-for-profit.

Mark Smith: 18:45
And when I, you know, I engaged them because I said, you know, I've got intro id because I wanted to validate where their mail was right before we started, it was an old friend who I hadn't seen for years and so I didn't know how they're running their charity. It's a successful charity and what they're doing in the community and I was his email domain, everything onto M365, bundled that up and I was thinking, oh my gosh, did he get non-for-profit licensing? But at the end of the day, the cost for one year, for the number of staff they had, was absolutely negligible. And I think there's this perception that, oh, it's Microsoft that's costly For a startup. What you get bang for buck, as you say is absolutely incredible and a no-brainer. Because what I did notice is that after he left my place I haven't heard from him since, and that's a positive thing. That means everything's working right it just worked.

Mark Smith: 19:49
And I just, oh, you know, know, in the old days where you'd set up your apache server and your you know smtp servers and pop servers and stuff, and it's just like you'd hold your you know breath that it all worked right and I had not put myself in that head space where I see so many startups go, oh we're gonna get a google suite because it's cheap or free, and I'm like, do you understand what you get out of the box with Microsoft? In that respect, and, as you say, with Dataverse, you're right about the mid-market being the toughest space from a scaling perspective. So what I'd like to know is that. I know that at the Brussels conference and forgive me for not saying names, I can never remember how many letters and acronyms there are what was it?

Andrew Welch : 20:38
EPPC, the European Power Platform Conference. I think EPPC.

Mark Smith: 20:44
Fantastic. I plan to be there next year. But that aside, I saw the photos of the session and they were pretty much down the hallway. The room was absolutely jam-packed I think about 3,000 people in the room plus what flowed out the door. Now tell me what's been the response, as you've shared this message. What's been the response?

Andrew Welch : 21:08
The metrics on this and I'm going to get this almost right. I think that we had. So the timeline here was that we presented the white paper at Dynamics Minds and we shared it with those who attended Dynamics Minds. We then, only two weeks later, had EPPC in Brussels. Then, only two weeks later, had EPPC in Brussels, where we presented the session, the topics again, and we did. I mean, we had probably 300 people in that room. They people flowing out the door, and it was a wild room too. It looked like a room where a parliamentary committee would meet, and it was. I, I think I said at the beginning of the session that I always imagine that if I ever spoke on the floor of a room like that, it was going to be in order to get a law passed. But no, no, no, we were presenting on Power.

Mark Smith: 21:57
Platform in a modern data platform architecture.

Andrew Welch : 22:00
And it was on that day. The day of the session was when we released the paper and posted about it and shared it. I think the numbers are that we had over 500 downloads of the white paper in 24 hours. We had just dozens and dozens of reshares of the content. I spent several hours just replying to comments on comments and questions on LinkedIn. So I would say it was probably the single most popular piece of content that I had ever produced, at least in its uptake within the first 24 hours.

Andrew Welch : 22:37
So it's very clear to me that there's a lot of demand for the idea of how do we make Power Platform work with the rest of our data platform. How do we use Power Platform with our organizational data? And going the other way, how do we use power platform? If I'm a data platform owner? How do I use power platform and take advantage of this Microsoft technology that I might not be totally familiar with to really drive the data platform deeper into the daily work of my colleagues, you know, and the people that I work with? So the response has been phenomenal and I will call out and say that this white paper was written. I wrote this very, very much like very collaboratively with Microsoft. So this had eyes on it from folks that you know, folks that actually built this technology. So it's been really great.

Mark Smith: 23:34
Yeah, so we've obviously been inundated with one the people downloading it and, of course, the spinoff from that has been that a lot of people are asking we want more like can we, can we um partner with cloud lighthouse in our own customers? Can you guys come in and run sessions with our clients in a partner situation? Then, of course, we've had customers coming to us and asking about practically how do we do this for our organization? Right, and it opens doors for us to, of course, come in, run workshops, and so it doesn't matter where you're located in the world If this is something that you want to do. The one or two first kind of let's call it hand-holding with us. We you know, particularly Andrew has a service here that allows you to get in touch with us, and that's whether you're a Microsoft partner already or an end customer, and we can actually bring this to life in the context of your data estate, your plans. We can work with your enterprise architect group and really get that blueprint in place and a roadmap for your organization.

Mark Smith: 24:48
In context of this, andrew, what are you finding from the people that you've already particularly started doing this for? What are you seeing?

Andrew Welch : 24:57
Yeah, well, so this goes back to something you said a moment ago about enterprise architects. So we've talked a lot on this show before about the need to bring enterprise architects on board with this idea of ecosystem-oriented architecture, which I think in the Microsoft space, by definition involves power, platform, fabric, azure, data and infrastructure and security services, etc. So I think that we are one thing I'm really pleased about over the last, you know, four or five months or so is I think that we've done a good job and I'm starting to see more and more folks do this. So I'm thinking Carl Cookson, for example, launched a blog that you know. He's producing really good, deep content about this, really good deep content about this but it's really designed to help enterprise architects get on board with and give the thumbs up to the CIO in their organization that, yeah, this approach is legitimate, it's robust, it's scalable and it's going to pay dividends for us. So the first thing that I'm seeing as I'm now you know, I've got a number of workshops coming up where I'm taking on this topic right and helping organizations build their data strategy and architect to make that data strategy actionable the first and foremost thing that I'm seeing is a growing acceptance among enterprise architects that these types of approaches are the right way to go.

Andrew Welch : 26:26
We had a question actually, when we did our last episode of the show, we did a listener Q&A, so if you haven't seen that episode, go listen. But one of the really good questions, but something to be effective. I'm an ERP architect. I've spent a lot of time with Dynamics, f&o. I want to, I need to start building ecosystem-oriented architectures. How do I do that? How do I get started? And the thing that delighted me about that question right, is that I feel like, at least in my experience, the ERP architects are always the most skeptical, all right, are often the most difficult to convince that there's a new or a different way of doing things. So it heartened me to hear that question coming from an ERP architect. But I also think that you know we're just creating momentum and we're creating the material and the patterns and the reference architectures to back that up. We're just creating such good momentum around ecosystem-oriented architecture, which is you know, I think you know, anna and I definitely believe this is the future for most organizations of any size.

Ana Welch : 27:36
I would add that this also aids those organizations with traditional divisions of I don't know database engineers and then developers, and it gives these people a door to open towards ecosystem architecture as well a door to open towards ecosystem architecture as well. In fact, a lot of the audience that we've had in Brussels was not Power Platform. They were NET developers wanting to see okay, so what do we do now? How does a system look like nowadays and how does it all integrate together? Because you know, the same question comes up over and over and over again Dataverse is the most expensive database in the world.

Ana Welch : 28:37
Why are you guys using it? And the fact that it sparks, and the fact that it sparks curiosity, and this question in those people's minds is really important, because many organizations are still battling with creating a DevOps-type strategy, environment, culture where everybody collaborates and works together, where we're finding the right solution for the problem not the right solution that we know will work because we've tried it before and because it's the technology that we know, et cetera, et cetera. So I believe this is what makes the white paper so damn successful. It's because it doesn't just talk about the app and the Power Platform developers. It reaches everybody.

Mark Smith: 29:28
I like it.

Andrew Welch : 29:29
Let me ask you a question, anna. So one thing that we've not. We've talked about enterprise, we've talked about SMB, we've talked about individual technologists, so we've been very through this conversation, very focused on those folks, but we haven't talked a lot about Microsoft partners, which are obviously such an important part of this equation. So let me ask you and I have my opinions but how do you believe Microsoft implementation partners and consultancies should be taking the concepts in power platform and modern data platform architecture and making them real in their own work? What are your thoughts?

Ana Welch : 30:18
I mean, I think they should at least start, andrew, and even the what I thought was surely long dead lift and shift strategy. I mean, it's a cry for help. Partners have to start doing this because their customers are going to request it. My customers are requesting it and my customers are government institutions or public safety and national security, and you would think that they're stuck in the past and old-fashioned and in their jobs for 20 years or more and, whilst they may be, that they're also asking the right questions. So customers are going to start asking the right questions. How do I integrate X with Y? How do I make my data secure? And at the same time, by the way, I want to use Copilot?

Ana Welch : 31:19
So, partners are going to have to stop writing. You know proposals that start with. We're just going to model the tables in Dataverse like your Excel spreadsheet is. Yeah, totally yeah.

Andrew Welch : 31:36
What do you?

Mark Smith: 31:36
think, mark, I think I'd like to hear what you think, and the reason I'm saying that is that, as in, we're at time, because I wrote the damn thing. We're at time. So, as in, I don't want to go on too much longer, but it's interesting. What you said there, anna, is that I think, andrew, there is room for another white paper to be written directly, aimed at Microsoft partners and, how you know, the partner of the future really is going to need to operate in a new way, because I think things are changing so rapidly and I think that the end customer is becoming so much more mature and their expectations are going so much higher that the traditional partner is not delivering. And you know, I wrote that post a few weeks ago, which has been my most successful linkedin post of all time around. The microsoft partner is dead and I think and of course it was tongue in cheek, absolutely, and I didn't specify what partner sizing, but I think there's a pivot to a new type of partner and I think, as I say, andrew, there needs to be almost a white paper around this and of course, you're so good at writing these things to really go, because what are the questions come out, so what's next? So what do you do? How do you pivot what and you know, in the last 24 hours I've done multiple meetings with people in the UK that fit these profiles One new boutique partners starting up that are going. You know we want to set ourselves on the right trajectory. We just don't want to become a traditional partner as we scale.

Mark Smith: 33:17
And the second one is where I'm seeing in customers and this is another conversation I've had in the last 24 hours one of the largest, I won't say actually their industry in the United Kingdom, but it's a very large player over there that are moving away from their partner that they've been with for seven years and bringing the whole operation in-house around their data estate, around app building, around the services they will offer to their constituent, their customer base, their clients, and then also making the IP that they develop available to their peers in the same industry. And I'm seeing that pivot. You know, as a lot of people are like, customers will never be able to do this for themselves. They need a partner. And once again it just reinforced that you're getting in customers going.

Mark Smith: 34:08
You know what we'll run our own practice and why were they reaching out to us is because they want help on that journey of getting that structure in place and, and you know, one of their workloads is a major on-premise lift to the cloud and they want to do it right. They don't want to just do a lift and shift and same thing in a different, you know, location from their data perspective, but they want to go. Okay, how are we readying our organization for the future, where we know the IT landscape alone is going to go through a massive, monumental shift over the next five years? They don't want to just do business as they've been doing business, because it's not preparing their company for the future. So, yeah, take up that challenge, andrew white paper please it's almost like we discussed it before.

Andrew Welch : 35:06
but the only other thing that I'll add and then we can wrap this up because I know we're at time, but is that I would love to see some Microsoft partners take the paper and look at some of the patterns. I'm thinking specifically patterns three and four, so master data node and data landing zone and develop IP that makes those sorts of relatively complex architectures, to make that those architectures more accessible and to make them more robust more scalable.

Mark Smith: 35:37
I would love a exactly exactly.

Andrew Welch : 35:40
So that's my challenge to partners is to build some IP that that makes this makes this more accessible. I think you have a winner there if you do that.

Mark Smith: 35:50
Awesome. Well, thanks everybody. Please reach out. Hit us up on the LinkedIn posts, the X posts, when we publish this. Go to cloudlighthouse and check us out. Get in touch If we can help you. We want to cloudlighthouse and check us out. Get in touch If we can help you. We want to help you. We're not implementation partners. We will come along, we'll help you with strategy. But thanks guys for joining all of us from remote locations today.

Andrew Welch : 36:19
Thanks, thanks, guys See you.

Ana Welch : 36:21
Bye.

Mark Smith: 36:23
Thanks for tuning in to the Ecosystem Show. We hope you found today's discussion insightful and thought-provoking, and maybe you had a laugh or two. Remember your feedback and challenges help us all grow, so don't hesitate to share your perspective. Stay connected with us for more innovative ideas and strategies to enhance your software estate. Until next time, keep pushing the boundaries and creating value. See you on the next episode.