Scaling Microsoft 365 Copilot over 400,000 Users at EY with Tom Crinnegan

Prefer audio?

Who should listen: Microsoft 365 Copilot service owners, AI enablement and adoption leads, and anyone working out how to govern the Frontier Program and usage-based billing in a large organisation.

Guest: Tom Crinnegan — Microsoft 365 AI Services & Governance Leader, EY

EY has 420,000 people in over 140 countries, is all in on Microsoft 365 E7, and its client-facing teams want every new Copilot feature the day Microsoft announces it. Tom Crinnegan leads the team that runs Microsoft 365 Copilot, Copilot Cowork and the Frontier Program inside the firm. He walks through how EY keeps Frontier deliberately small, reaches 7-8,000 people a week with enablement, tracks the Copilot firehose, and ties usage-based billing back to the people who own the work.

Many thanks to Neat, sponsor of this episode.

Key insights

• EY has around 420,000 people in over 140 countries, so the team uses every admin control available to split Copilot policies by what each part of the firm is comfortable with, and spends a lot of time with data protection and risk on what to enable. ▶ 2:55
• Being on Microsoft 365 E7 removes the licensing component arguments, including Agent 365 as it matures. Crinnegan reckons usage-based billing will introduce a new layer of haves and have-nots in most companies: can you justify what you want to use the tool for and show the value? ▶ 5:13
• Frontier access at EY is kept deliberately small. People complete a form, are told there is no formal support or service desk for those features, and in return help the team understand them in a business context. Crinnegan warns them not to get wedded to Frontier features because some may never reach general availability. ▶ 7:13
• Feedback to Microsoft is harder than it sounds. Microsoft wants logs and prompt flows, but EY does not allow people to send feedback that includes their prompts, so the team tries to replicate issues itself without EY data, with Work IQ turned off. ▶ 9:41
• The Frontier sign-up form is never advertised; curious people find their way to the team through a Teams channel. GA features get the opposite treatment: a weekly What's New and Next session that drew 7-8,000 attendees the day before recording. The service adoption team is kept on the GA track, and anyone who needs to see Frontier gets a separate non-HR service account on its own virtual machine. ▶ 11:04
• The team has been building training videos with Copilot Cowork and a couple of skills: take 15-20 screenshots of a feature, hand them over, and ask it to create the video. When the UI changes they swap a screenshot and rebuild rather than re-recording. ▶ 13:34
• Questions in EY's Viva Engage Copilot community are answered by a Power Automate flow with AI Builder rather than an agent, which gave the team control over the response format and a confidence threshold. Low-confidence answers are funnelled off for review, so the team can see the 15 questions it was unsure about each month and update the knowledge source. ▶ 16:01
• Copilot touches every Microsoft 365 workload, so the team runs Teams channels with every product team, including one for agent governance that spans Power Platform and InfoSec, plus fortnightly feature update calls for risk, data protection and AI enablement groups in every region. Crinnegan describes the team's role as an aggregator: people do not need to know who to ask, they come to the forum and get routed. ▶ 20:04
• A homegrown Power App pulls in Message Center, the Roadmap and even blog posts, since some things are announced without a Message Center post, so the team can see what has been looked at, descoped or handed to another team. Minor UI changes get descoped outright because there are 20 other things to look at. ▶ 22:59
• Usage reporting moved from CSV exports to API ingestion into Power BI, with role-based access down to a service line in one country. On top of Microsoft's data the team defined KPIs for habitual use, and saw it jump as the models improved. Tom's take: Copilot is a very different story from two years ago and worth a revisit. ▶ 26:28
• Copilot in Excel went from "I can't do it, but I can tell you how" to giving Crinnegan four approaches to splitting a user base into deployment tranches, two of which he had not thought of. EY now brings its own Excel world championship competitors into training content, because when they say Copilot in Excel is worth it, people listen. ▶ 28:23
• EY's usage-based billing process requires people to request what they need and ties it to the lowest level of budgetary control available, down to the internal charge codes already used for people's time. Copilot Cowork has been onboarded onto EY's existing internal marketplace of technology services. Tom's observation: professional services firms are culturally suited to this because they already bill by client code. ▶ 30:54
• Reporting is still a pain: the API-level data is not where it needs to be, and pilot users complained the Power BI report was two days out of date, so the team pointed them at /cost in the app. Both expect API coverage to follow because the demand is there. ▶ 32:18
• The latest EY-Parthenon CEO Outlook Survey of 1,200 CEOs found most have visibility of AI spend but only a small minority have strong control over it and clear ROI insight. Crinnegan's conclusion: tie every request for a usage-based service back to the person who governs the work, or you cannot show value. ▶ 34:20
• Tom's metaphor: the manager of a five-person marketing team is the only one who knows whether $1,000 on Cowork delivered a product fast or just summarised someone's inbox every hour. Crinnegan agrees that clearing 60-70% of a mailbox could be the best $30 or $40 ever spent if it frees someone up to answer more RFPs, which is exactly why IT cannot judge it. ▶ 35:20
• Copilot Code and vibe-coding tools are bringing people with no development background to the point of building something, then hitting the governance wall: evals, regression testing, model drift and what happens when a model is deprecated in two weeks. Crinnegan has spent more time with the Power Platform and Dynamics teams in the last six months than in the prior three years. ▶ 40:37
• If an agent is business critical, someone in that team needs to own consumption management: understand what it consumes, set up alerting and look at projections. Start small, because if you give something to 20,000 people on day one, three of them will burn through all the credits. ▶ 46:25
• Tom's view: this is the business's budget, and IT's job is to facilitate and govern, not gatekeep. EY's business is already used to pay-for-use from Azure services, with finance business partners showing exactly where the money went. Both agree a chargeback model becomes unavoidable once one country goes further on AI than the others. ▶ 47:22

Insights summarised by AI from the episode transcript, reviewed by the Empowering.Cloud team.

Listen: Apple Podcasts · Spotify · Other platforms