Name: Launching Winning Features Every Time with Mixpanel Experiments
Uploaded: 2026-01-14T11:16:14.989Z
Duration: 50 min 12 s
Description: Launching Winning Features Every Time with Mixpanel Experiments

Transcript for "Launching Winning Features Every Time with Mixpanel Experiments": Good morning, everyone, or good afternoon. We will just let people join today. We've we've got a quite a few people that have signed up. So we'll just wait for everyone to come in two or three minutes, and then we'll get started. Yeah. Where's everyone joining from today? What countries? Pop it in the chat. Welcome. Welcome. South Africa, Italy, Germany, all over. Oh, wow. Fantastic. Great to have you all here. So it's definitely definitely afternoon time for a lot of you. Mountains of Norway. Good stuff. Fantastic. One or two more minutes, everyone, and we'll get going. Thanks for your patience. Alright, everyone. Let's get started. I wanna be respectful of everyone's time as well. So let's go ahead and share my screen to kick things off. And there we go. Okay. Right then, everyone. Hopefully, you can all see my screen now. Good morning. Happy New Year to everyone. It probably is a bit strange saying that. It's probably the last time I can get away with saying that, actually, in the January. But happy New Year to you all. Really excited to welcome you to our session on launching winning features every time with Mixed Panel Experiments. Hopefully, that that was what you're expecting today. So we've got about sixty minutes here today to cover things. We can we can kind of take our time. We're gonna explore primarily, of course, how modern product teams can dramatically increase the odds of shipping features that actually move the needle, not by guessing, but by validating every step by using data. Today's session is designed especially for product and engineering teams, as well as marketing, design teams, UX teams as well who partner closely to ship things like customer facing experiences. And, of course, our analytics teams as well who are the backbone of the the whole data driven decision making. So great to to have you here as well. My goal today is to show you how experimentation now sits at the center of building products within Mixpanel. I'm gonna walk you through hopefully, you'll find this useful. I'm gonna walk you through a very practical narrative. I'm gonna walk you through how Teams are gonna move from insight, hypothesis, validation, and finally measuring impact all within a single platform. I'm gonna ground everything in quite a clear relatable use case so you can see exactly how this is gonna work end to end. So by the end of today's session, my aim is for you, you know, that you can walk away from this this webinar today with a clear understanding of why should analytics and experimentation live alongside each other, why it matters in today's quite fast changing product landscape, We all know that. And how you can start using it to launch features with confidence and and speed. So little agenda for the session. Before we dive in, this is this is kind of really what we're gonna cover. I am gonna start with a short primer on why analytics and experimentation belong together. It is just a few slides. Please believe me. Everything is really gonna be in product today. I do want I do need to set the scene though with a couple of kind of background elements here. It's really the foundation for understanding why teams combine combine both and really able to ship better features, essentially. Then we're gonna spend the bulk of our time, about thirty minutes or so, walking through a live demo. This is the the fun part for me. I get to show you actual Mixpanel real live product. This is where I'm gonna show you how Mixpanel supports that entire cycle of, like, identifying an insight, forming some form of hypothesis, running an experiment, and finally measuring the impact. That's really the end to end that I'm gonna show you, and it's gonna be that practical end to end use case. And we'll wrap things up with a q and a. Couple of housekeeping things about the q and a as well. There are some preloaded questions that a lot of you have sent prior to this registration. I'm gonna focus on answering one of those at the end. My esteemed colleague, Amit, is on the call today because there's a lot of you. And as you're walking through as I'm walking through things today, I he will be monitoring the chat. So there are two sections of this this GoldCast webinar. You've got a chat, you've got a q and a section. Feel free to ask the q and a in the q and a section. But if you pull it into the chat, we will review those and they might move it over to the to the q and a section as well. So do look at both toggles to see whether your question has been answered. But there are a lot of you on the call today, so we will do our best to get everything get around to all of your questions as well. And there will be some follow-up as well that you can ask additional question after this webinar too. Okay? Let's get started. Sort of introduce introduce myself at the start, really, I guess, but I'm, Max, I'm a solutions engineer at Mixpanel. I've been at Mixpanel for three and a half years now. I work across product and yeah. Product and analytics ecosystem, helping teams build data foundations using Mixpanel, running better experiments, shipping features with confidence. It's a really cool role, actually. I get to work closely with the likes of yourself, whether it's product engineering design teams, marketing teams in in our EMEA function. I guide them through what the sort of modern analytics era sort of looks like. It's an exciting space to work in. And, yeah, it's really it's been really fun so far. Only bit of audience participation I'm asking for you here now is is a poll. Right? And my colleague is gonna gonna pull up a poll question for you. It's just really useful for me to be able to gauge, you know, the audience today in terms of what you guys are, what what everyone's situation is. So, Larissa, if it's okay, would you be able to pull up the poll question up for us? So you should see this pop up on the screen. Okay. So the poll question is you should have it now. Right? You should have it, everyone? Oops. Yeah. Yeah. The poll question is which let's go to the polls tab. Which statement best matches your experimentation culture? We trust our instincts. We we test when there's time. We test important decisions. If it's not tested, we it doesn't ship effectively. So, yeah. It's just really useful for me to get an understanding. So the poll is also available in the in a in a tab as well. Great. Okay. It's actually quite equal, though, but it seems like most users or most of you today are saying, like, we test important decisions, which is great. So it sounds like the tell me if I'm wrong here. It sounds like the majority of you on the call today, you know, will, you know, will have to do or use some form of experimentation process at the moment, whether it be through another tool, whether it be, like, you know, you you test kind of manually at the moment, you know, or also at the same time, it looks like we test when there's time. That's a that that's an interesting outcome as well. So thank you very much, everyone. That's that's very useful for me to understand in terms of, like, what everyone is looking to do or what what everyone's current status is. So today, innovation moves kind of faster than ever. Right? Competitors pivot quickly. Features do evolve constantly, and your your users' expectations can shift literally overnight. So in that kind of environment, it's a tough environment, teams can't afford long cycles between an insight, a decision, and some form of action. So to truly build at the speed of innovation, teams like yourselves need this kind of continuous loop, not a disconnected tool or a disconnect or or a, like, a silo tool. Right? Analytics will help you understand what's happening in your product, but experimentation, of course, will help you validate what you're gonna do next. But when these two functions live in different systems, which is what we see quite commonly, the three these three big things start to break. Firstly, the metrics that match across tools. Right? That can be intensely frustrating. Lots of teams end up debating numbers instead of actually making decisions. Secondly, the analysis becomes very fragmented, and it becomes very slow. You're jumping between dashboards. You're jumping you know, you're looking at exports. You're doing spreadsheets. You're doing things manually. And experimentation tools sorry. And then a separate experimentation tool is used just to answer those sort of simple questions. Right? And thirdly, the most importantly one, insights don't translate into action quickly enough. Right? So by the time you've aligned on data, you run an experiment, and you get it running, the opportunity may have moved, you have to shift to something else. And this is exactly the gap that our experience is designed to close by bringing analytics and experimentation together so that teams can just move from an insight to an action in a single kind of workflow. That's what we're gonna walk through today. But here's the shift that we're seeing. Right? See, we are seeing quite a a significant market shift at the moment across, like, typical product teams. They're no longer treating analytics and experimentation as two separate workflows. They're bringing everything together into a unified data power loop data powered loop, should I say. And and with mixed panel, experiments run on the same events. We run on the same cohorts, all the same trusted metrics that you have. So that means you don't have to export data. You don't have to rebuild KPIs in another tool. And there's no drift between what analytics says and what your experimentation tool says as well. So everything sits on the single consistent foundation. When you unify all these pieces, your workflow becomes this straight line that you're hoping for, insight hypothesis, test, validate, and impact. So you're not switching systems. You're not reconciling any numbers. You're you're literally moving forward at the pace that you need to in this kind of landscape. So the result is innovation at the speed of, say, insight that you normally get through analytics. And when they live together in one data layer, the iteration is gonna become faster. Your data is gonna become cleaner. Decisions are gonna be more confident that you're gonna make, and every feature release becomes measurable from from day one. Let's let's go into a demo in a second, but just to just to ground it in a real life example of how experimentation and analytics work together. STEP, for those of you who know STEP, they were one of our early beta users of Mixed Panel Experiments. They're a digital bank. They're designed especially for teens and young adults. And one one of the things they really had to do when they tested our experiments was to appeal to a young generation of tech savvy users, right, Gen Z, whatever it may be, with a user experience that set them apart from all other banking platforms. They wanted their users, their young, you know, audience or their young user base to to use Step as their primary bank account. And to drive that shift, they used kind of our analytics and experimentation together. So they they ran quite a few like, a series of deeply informed analytics powered experiments. They identified the exact moments, behaviors that were key to creating sticky users that eventually may step their primary account for their user base. So they use those insights to test target feature changes and nudges, things like that. But the result is the important part here. There was a 14% uplift in customer setting direct deposits, which ultimately meant that they were, you know, setting Step as their primary bank account within the first thirty days. And more importantly, Step didn't just get that lift in one metric. And they built this culture of experimentation where every idea is validated, every release is measured, every team was able to move forward with clarity. So real life use case for you there. Brilliant. Let's get ourselves into a demonstration now for you. And this is where I'm going to kind of ground everything in that kind of real life example. Now I just have to kind of set a caveat here. Of course, this is synthetic data that I'm using today. So I have to just imagine a few things. I have to use my imagination to to play the role, say, of a product manager. And in order to do that, I've got a product, and this product is called WeBuy. It's my it's my ecommerce storefront. So I'm just gonna imagine that that this is my business. Right? When WeBuy, it's an ecommerce business centered around selling multiple products ranging from tech to fitness, home products, whatever whatever you wanna buy, you can buy it on my storefront. Right? And I'm a product manager, let's say, and I want to see how I can improve average car value for all my items. Right? That's that's kind of what I focus on in terms of my key my key KPI. So in order to do that, I'm actually tracking everything in Mixpanel. And the first thing I'm gonna start off is go into a Mixpanel project with where I'm tracking all of my Weebie data through events. K? So all of my WeBuy data comes into Mixpanel. I'm I'm actually looking here to start off with on a metric tree. And my metric tree really is my overall representation of the key drivers for WeBuy, my ecom business. Right? It's something that I can use in management meetings to understand what our key KPIs are, what drives our top line. In this case, it's purchase revenue. Right? You can see here it's purchase revenue, which is all built from event based metrics that we capture from user interactions. Right? We're building these into nodes, and we've got, like, a correlation here between all of these different metrics that we measure. That's really useful. That allows us to have that kind of top down conversation. But importantly as well, everyone's an owner of these metrics. I've got a colleague here that owns this purchase conversion metrics. I've got all my secondary metrics here that lead up to purchase revenue. But importantly, what we're looking at here today is average cart value. Right? Average cart value is the thing that I'm interested in, that I'm the product manager for, and I own this particular node. I own this particular metric. Right? And you can see here, I've got my purchase completed as average items price, right, which is the query that I've run to look at average cart value here. Now importantly, in order to try and drive my average order value up, I'm gonna start to run, hey, presto, an experiment. And you can see here, when we click into each of these nodes in our metric tree, I've got average cart value here and a bunch of stuff that I can look at. I've actually run some other experiments as well already. So there might be some existing experiments that we've run to try and drive average car value. I'm gonna start more from scratch today so you can see how it works. We've got some pin content. We've got a purchase funnel that we're looking at related to this average car value. We've also got a logbook entry as well. And this logbook entry is here saying, I'm gonna run an experiment from January 2026 on different voucher types to drive my average cart value. That's what I feel is gonna actually improve average order value. Right? So in order to do that, we could go to our experiment section here. We can actually create an experiment directly from a node, right, from this node. Or actually, you can see here if we just zoom out a little bit. There we go. On the left hand side of our screen, we've also got experiments and feature flags that sit here in the management layer on the left hand tooltip bar. Right? Feature flags, we'll cover in a second. Right? They're kind of intrinsically linked. We'll kind of set them set them aside set them into two separate kind of, like, elements here. Alright. So let's go ahead and run an experiment. Let's just go into our experimentation hub. This is this is what we're looking at here. We're looking into experiments. And let's say, like, I'm this this user here. This is my experimentation, like, board. These are all the experiments I'm running at the moment, and you can see what the status is of all my experiments. You can see here I've got a coupon offer on testimonials experiments that are either in progress or just being configured, and some of them that we've made decisions on with a high level primary view sorry, a view of the primary results. But let's start a new experiment from here so you can see what the end to end process looks like. So we're gonna click on new experiment to start with, where I can then use Mixpanel's feature flagging to start to deploy a feature flag to those users or specific variants here. Now it's important to note as well that we are we are actually agnostic as a tool. So we we can send we can allow you to actually send exposure events by leveraging third party feature flags too. Right? So if you if you wanted to if you use, you know, another experimentation tool where you already deploy experiments, you can actually use that as well. But in this case, today, I'm gonna use mixed panels feature flagging. And like I say, today, wanna start to deploy some, like, different voucher codes to my users to see whether that that's gonna have any impact on average order value. So I'm gonna call this new voucher experiments. Right? And that's gonna be the experiment name. But importantly, when we're starting to configure this experiment, that's gonna be the feature flag key that's being set up. This is what Mixpanel's API is gonna call here to determine which user is is going to receive which variant that I expect that I that I'm gonna set in my configuration for the experiment. Right? I'm also gonna add in a hypothesis here. So I might say as a product manager, I believe that adding voucher codes on our storefront will increase, average on value. Right? Or whatever it may may be. As you can tell, pretty terrible at typing and talking in demos. Right? So we'll go ahead and create this experiment. There we go. This is our new voucher experiment that's that's just kind of, like, just been created here. So we can see and this is the home screen. This is kind of like that the dashboard or the main hub for this new experiment that we're gonna run with our hypothesis. We haven't got any exposures yet because we haven't configured the experiment, which is what I'm gonna do. And you can see here we can start to add our primary and guardrail and whatever metrics we need. Right? So let's go ahead and configure this experiment. And this is one of the really useful parts of Mix panel. Okay? Everything is in one platform. So today, what I'm gonna do is, like, measure a primary metric or my goal, right, for this. And my goal or my primary metric that I'm measuring is gonna be average cart value. So you can see here from all the event driven metrics that we build as a business with our in our mixed panel project, we have total clarity of of everything that we're measuring. So our average cart value is a metric that we save across our mixed panel project. Everyone can be that those can be reused within experiments. So everything within one platform, trusted, same similar metrics that everyone measures, are all part of experiments too when you use Mixpanel. We're also gonna set a test duration. So we can see here, we've got 10,000 exposures, or we can switch to days for the test duration. But importantly here, a lot of customers ask us, like, how do we choose a sample size? How do we know how many people to sample depending on our business size? We even have have a sample size calculator that allows you to do that. So maybe we'll choose, like, a baseline conversion rate. So at a minimum, we expect, like, a 20% baseline conversion rate. What's our minimum detectable effects going to be? And maybe it's, like, 15, for instance, or, like, 20. All of this that has, by the way, all the descriptions around how the calculator works. You know, we might want to say, well, we're gonna actually expose users to four different coupon codes or four different variants and set our confidence level. This allows us to then determine, like, how we want our or how many samples we're actually going to leverage. Right? So maybe it's just gonna be 20 let's just say 25 or actually maybe 10%. Here we are. 10 is ours. It's our baseline conversion rate. So you can see here, it's good. Tinker with your sampling size effectively based on the calculator. So it's a really useful tool that you can leverage. Let's keep it at 10,000 exposures for now. In my advanced settings, this tells me what type of test I want to run. So the statistical significance will come from the type of test that you decide to run-in the experiment. So is it gonna be a frequentist type test, which is really of a a model which is capable of detecting a smaller lift, but you do have to run the experiment for the full duration? Or am I gonna run a sequential test, which is actually more of a quicker experiment, and it's gonna allow to detect lift, but it might not reach the statistical significance for the smaller lifts that we're looking for. But because I need to move quickly here as a product manager, I wanna release voucher codes quickly to try and drive revenue, try and meet the goals that we need. I've gotta run a sequential type test. And then we can choose our confidence threshold here as well. Let's continue on configuring my experiment here. So in my customization guide, I can start to add my variants. So maybe I want to add, like, multiple variants here. So you can do simple a b tests, like a test a b test. You can add just a feature gate where you can switch a feature on or off by using feature flags. But in this has been but in this instance, I'm gonna add, like, four variants. They're just gonna be simply called today a, b, c, and d variants. Right? And these are the different voucher codes. We, of course, have that control variant that allows us to understand what the lift is going to be. Let's choose some rollout groups as well. And again, this is the really useful part of leveraging Mixpanel. When you when you choose rollout groups, this allows you to determine who's going to view or who who you can target for the feature flag. Now many other experimentation tools just say, just fire them out to, like, x amount of users. Right? When you choose a sample, it'll just go to certain users. But with Mixpanel, you could use all of the event driven methodology or data model that we have to target specific users that you're interested in for the type of experiment that you're running. So for voucher code experiments, I might wanna do a couple of things. Right? I I've already built cohorts in Mixpanel. So for those of you that use Mixpanel today, you'll know what cohorts are. They're kind of first class citizens in Mixpanel. Maybe for voucher codes, I'm not really interested in running new users or looking at new users for this experiment. Right? So I'm gonna exclude new users from this experiment. I'm also going to exclude users from other experiments that I've already been running. Right? Because I don't want to be part of, like, multiple experiments here. I'm actually gonna exclude them from this a, c experiment. I'm gonna say is not set. But, again, as mentioned, you can use all the events that you capture by mixed panel tracking to start to target users that you're interested in for this experiment. So for voucher codes, I wanna target users that have viewed products greater than or equal to once in the last thirty days, where the product category is, should we say, accessories, books and media, electronics, and home and garden. K? And those, like, are gonna go up to, like, my my specific variants. Right? Now because it's dummy data, as you can see here, there's no users in that cohort. Right? So what I'm gonna have to do is actually just remove, like I think it's these two cohorts here that actually gives me something. Right? Now in reality, I've got so I've got 627 users in that cohort that are gonna be targeted for these for these variants in this experiment. In reality, you'll have thousands of users in this cohort, but this is what fake data gives me sometimes. Now I can choose my percentage rollout. So let's say if you did have, like, 8,000 users or 10,000 users as part of these cohorts. Right? Maybe you want to only set that to 10% of the user base. Or do you want it to go up to a 100%? Or do you want it to go up to 70, oops, to 70% of our user base? Right? You can choose your rollout of percentage of users from the cohort that you've that you've selected. Let's just say today, wanna send out to, like, 20% of my user base. And then you can choose your variant split. So is there an equal split between variants, or is there kind of, like, a higher control percentage that you want people to to be exposed to or a higher variant split for for other types of variants here? And then we can click done. Okay? So we've now chosen and configured and finished and saved our experiment off here now. Okay? Brilliant. So we configured the the main part of the experiment that we need. K? All done that in the UI. Importantly, the things that we need to measure for this experiment are gonna feature as you can see down here. So we've got primary metrics, we've got guardrail metrics, and we've got secondary metrics that we can measure as part of this experiment too. So let's have a little look here. What we what have we already got? Well, we've already got average cart value, which is that common metric that we measure in my mixed panel project that's already available for me here. So we can start to see what things like the lift are and the control and the treatment are. And, you know, when the exposures happen during the experiment, we're able to understand or interpret the results. I might also want to add a guardrail metric here as well. And my guardrail metric, for instance, here is going to be from a funnel or an existing funnel. My guardrail metric is something that I don't effectively want to kind of be to change. Okay? I don't want my average time to convert for users through our purchase funnel to actually go down at all. Right? And if that happens, that needs investigating thoroughly, and I need to either abandon the experiment because we don't wanna lose our our users. We don't want our purchases time to convert to to reduce. I don't wanna start using losing our user base as a result of running an experiment like this. So this is where you can set these guardrail metrics to ensure that users sorry, that you're you're monitoring the safety aspects of the experiment as well and either defer decisions or abandon if you need to or further diagnose from there. And I'll also add my secondary metric too, which is, of course, gonna be that purchase funnel. So this is like a pre pre created metric that, again, that I've used in Mixpanel, which is my my classic ecommerce purchase funnel. I use a few item listings all the way through to completing a purchase within one session. You can see here that I'm measuring conversion rate. So cool secondary metric to be able to measure as part of this experiment is, you know, what is my conversion rate going to be as part of the experiment? Brilliant. Couple of other things to note. Like, we've configured most of the things that we need for this experiment here. If we go back to our settings, you can see here this is the feature flag that we have set. Okay? And if we actually open this up in another tab, you can see here I mentioned this at the start. Feature flags and experiments are kind of like almost kind of two two separate kind of entities here. This allows you to leverage your own tools for feature flagging and then send exposure events to Mixpanel, or you can deploy feature flags directly from our tool as well. Okay? And you can see here now this new voucher experiment flag that I've created provides all of the variants that I've already configured. And when the exposures occur, when we deploy this flag, this will tell us where the exposures happen. Developers and engineers, we have you covered. You can be added as QA testers within this part of the the feature flagging configuration as well. So we may have a couple or two or three testers to be able to gate our experiment or this flag to ensure that everything is as it should be prior to running this experiment. So all your QA testers can be added here. You can also see here the roll up groups that you can set to, which have already come through from the features like when I deployed it when when starting or creating the experiment. That's just kind of one thing to note here. And you can see the assignment key is our user ID at the moment. So those are going to to to kind of individual sorry. These are going out to distinct IDs. So the flag is gonna be deployed to users. Right? Brilliant. Let's go out of our feature flag for now, and we'll go back to our new new voucher experiment. And we'll maybe go ahead and launch the experiment. Now the sad reality is, of course, this is this is a kind of synthetic environment. And, unfortunately, I wish I could sit here and show you the results from this exact experiment that I'm running. But, of course, as it's in progress, we'd have to imagine that we're gonna wait, you know, a few weeks in advance in order for us to determine the results. So we'll have to go to another example experiment to show you how you would interpret results. But let's go into back into WeBite. So now we've configured the flag. We start the experiment. And when your app calls that Mixpanel's feature flags API, what we do is we evaluate the flag in real time using the user identity. We use the context. We use the rollout logic that you've applied in the slide, like, which cohorts. And we then determine it deterministically return I can never say that word. Return a consistent variant for the user. So you can see on we by now, we've deployed these different voucher codes you can see here. So as a user, I'd maybe get be getting exposed to one of the variants that you can see here, be able to reveal a coupon, and then use that copy of the code, and we'll see whether that has any increase in the metrics that we're looking to run. K. Perfect. Let's go back to our mixed panel project. And now let's look at a post experiment. Right? So we're gonna fast forward maybe two, two and a half weeks or however long it may may be once this experiment has has been has created some exposures as you can see here. So we're looking at the you know, the hypothesis and this this kind of post experiment like instance here. Well, we have some early significance. We can see the exposures for, you know, most of my variants, and then we can start interpreting the results here. Right? The primary metrics are indicating something quite significant that I can see. I don't know whether you can see that. Let's just zoom in a little bit. Variant b has given me a 37% improvement compared to the control variant, right, on our average order value. That's really good. That's a good start. And what we can see here from indicating off we're hovering over the other results here is if you can see that as well. Just hover over and zoom in a little bit. There we go. You can see here the p value showing at more or less zero as well. So we're extremely confident that this result isn't down to random noise or chance. Okay? So that's really good. This is a good good indication that variant b clearly is the winning variant for our average car party, which is the primary metric that we're measuring. I'll come back to guardrail metrics in a second because there's something to note there. We also have secondary metrics that we can measure too that you can see here. So, actually, variant b again is conclusive. It's giving us a 154% increase on our 154% lift short sorry, should I say, on our comment on on our overall conversion rates that we can see. Same with add to cart rate. Right? We can see kind of like a lift here on potential variants, but something looks pretty conclusive here. Like, variant b seems to look like the winning variant. However, there is a guardrail metric challenge that we can see. Variant b actually is showing us a 30% decrease on the guardrail. Right? Which was, if we look back at the metric, it was our average time to convert. You can see that. Right? Average time to convert through our funnel has gone down by 30%. That is something that might need investigating. So at this stage of an experiment, you might want to restart the analysis, make a decision, or defer your decision at the time, right, or abandon the experiment from here. Or you might want to do further analysis, which is what I would suggest. Right? The cool thing about Mixpanel is because everything's in one platform, you could look into time to convert a little bit more in a more of a an in-depth way. So what we're doing here is we're looking into that existing report in Mixpanel's core, like one of Mixpanel's core reports you can see here. Now let's just zoom in a little bit. This is, of course, like, some historical data from, like, you know, last year that we can see as an example. But if we look at average times convert, we're breaking it down, of course, because it's come from the experiment by all of the different variants. So if we focus on variant b here, what we're looking at over time is our average time to convert. Now we can see here there's a sudden spike of five point six hours. Right? Users are taking a long time to convert through a purchase funnel when they've been exposed to variant b. This is cool because that can actually start to investigate this further. So we could do things like view replays or do root cause analysis, which with this big dataset don't have any replays in this particular instance, so I'm gonna show you what that looks like. But you could create meaningful cohorts. Again, root cause analysis is part of this too. This allows you to do extra additional analysis to dig into the why and the how things have happened. But let's show you some examples of what replays look like. This is what you get when you view replays. Right? So we're looking at a replay here of our We Buy page, right, our storefront. And we also get indications at the bottom that you can see here. So these are all the all users maybe who have, like, had a high time to convert a particular time. We can now see where rage clicks happen or indications of frustration signals. And these are inferred within the replay at at specific times. Also, in relation to all the all the details of the activity that happens, so we can see here within the replay, all the events that get tracked that are inferred within the replay. So you get a real understanding of collectively what most of your users are struggling with. Is there a bug? Is there something we need to fix as a result here? Also get AI summaries and replays too that help you understand, well, okay. They viewed multiple products. They added an item to their cart. There was some frustration on dead clicks at certain stage, etcetera, etcetera. So this allows you to do things like copy replay links at a certain stage. Looks like there's a bug here. Maybe we need to fix this as a coupon bug or, like, when they try to apply it on the cart when they're trying to make a purchase. Let's send it to my developer. Let's get it fixed. Right? This allows you to do that extra kind of diagnosis and analysis. I think just another point to note as well is that when you're measuring something like a funnel, you can always break it down by experiments too. You can see there's an extra experiments breakdown now in Mixpanel. So you're allowed to ultimately break any ad hoc reports that you build by the type of experiment that you're running. So if you if you wanna look at a key metric, if you wanna do any additional ad hoc analysis, they become filterable and well, they're there with the ability to break down within, like, core reports as well. Great. Let's go back to our dashboard here. And what we can do is we can make a decision on this. Right? I wanna make a decision on my which variant to ship. K? Maybe we've actually fixed that bug. And we wanna say we wanna eventually ship it now. Now that we've stabilized that guardrail metric, you can defer decisions. You cannot ship, but you can choose to ship a specific variant here for all of your users and then document your decision. So we've said here significant boost to average order value and conversion rate. Bugs with coupon application have been fixed and rolled out. So we're all sorted. We're all set to go. So the kind of wraparound for this part here is that we can we can go back to our metric tree. Right? The the thing that we were looking at at the start here. And now we can go back to average cart value and start to obviously use the you can see here that we buy demo completed post experiment tells us that the uplift uplift has happened. So we have knowledge of this. We can add additional logbook entries as well. So I'll add another logbook entry here and just say, variant b was shipped most statistically significant, fixed the bug as well. Perfect. There we go. So we now have clarity of what's going on. This is great. Let's recap it. Okay? Teams manage this entire experimentation process in one place. You define rollout features with flags. You run experiments with deterministic exposure. You measure the lift against their metric tree that you can see here, and you're able to ship winning variants with confidence, all while measuring things like revenue impact all in the same platform. So I thought I hope you found that useful. That kind of completes the the demo part of the session. So what I'll do is I'll just stop sharing my screen, and we'll come back and just start to review some questions as well in a second. Okay. So, yeah, what I'll do from here is I'm I'm just gonna review. So a lot of you sent questions through prior to this session. So there's a couple that I think are really quite useful for those that are in the audience today that that kind of wanna understand a little bit more about experiments. So I'll just read through the questions, and I'll give my kind of response to to what I think is kind of like my best answer, really, based on what has been asked. There was a question here which says, what is required from our engineers to enable proper use of experiments? Is there an MVP that we can start with to show results quickly? So I mentioned this briefly, but to start using experiments, the the the the lift can actually be very small. The MVP is very simply you assign users a control or a treatment variant when you're tracking when you're when you're performing tracking, and you send an exposure event to Mixpanel. It's the exposure event is experiment started. You call you give it a name, experiment name, and then you give it a variant. K? The event should fire once per user experiment, and the user's variant must remain stable throughout the test. Right? So you can calculate lift. You can calculate confidence. You can look at impact. You can look at your existing product events, and you don't need Mixpanel's feature feature flagging system to do that. You just send experiment exposure events to Mixpanel, and you can leverage all of the the cool reporting functionalities we have. Another question here. As a start up, how do we balance what to AB test versus when to just ship and optimize without losing speed? And when should we close and clean up an experiment? Right? Well, as a start up, you should AB test your you should really, I would say, only AB test high impact or uncertain metrics. Something with, like, measurable changes, especially those tied to, say, a core metric, like activation, retention, revenue. Right? If the change is low impact or obvious, so you don't have enough users for for reliable results, it's normally better just to ship or learn qualitatively. Right? So if you wanna maintain speeds, keep the experiment simple. That's my advice for anyone that's, you know, using Mixedpanel's experimentation as a start up. Like, don't over optimize any tiny effects. Experiments are are really for the big decisions, not like pixel level type tweaks. And then I would just close experiments once, you know, all your predefined criteria met, You you know? Not based on, like, an early spike. Make sure you run the experiment for the full duration. And if the the variant clearly wins, then roll it out. Another question here. How do experiments work in an enterprise intranet or b to b context where users are internal and the user base is small and the adoption is low? I think it's a similar answer to the startup piece. Right? Experiments do work in the b to b kind of space as well, Internet products. I think it's just a different mindset for to sort of b to c. With a smaller user base, you're gonna typically expose a larger share of your users to experiment. Right? And because the sample size is limited, you need to prioritize experiments with, a big impact rather than, again, the micro optimization piece. And it's okay if the results are more directional than, like, perfectly statistically significant because you have a, like, you know, a smaller user base. So if it's directional, that might be okay. And, again, link everything to adoption, activation, depth of usage. Right? I think that's really important. Couple more questions to to cover here. How can we handle consistent metric definitions and governance when setting up AB tests? Well, the key is to treat, again, the metrics that we measure. We we showed that in the demonstration today. Treat treat metrics as the shared infrastructure, not something that you redefine per individual experiment because you will run a lot of flags. You will run a lot of experiments. You've gotta define those core metrics as saved metrics in mix Mixpanel is is my opinion, or some sort of behavior that you resave and reuse. Yeah. So standardize those metric roles upfront. One primary metric for success, a small set of guardrails that that shouldn't really regress, or opt option resecondary ones as well. And I would tie the primary metric back to a north star, like, or a metric tree that I showed before. It's a it's a experience. Clearly, ladder up to the company goals that you have. But do enforce governance across the process as well. So make sure you assign owners to key metrics. Make sure that everyone has a full documentation or definition of what you guys are looking to test so there's complete clarity in the experiment that you run. What are the key factors to get right in Mixpanel when launching a greenfield experimentation capability for scaling organizations? So I think here, like, yeah, if you're running experiments for the very first time, foundations matter more than the volume that you're testing. Just start with clear goals, clear hypothesis. Tie it to a north star. Tie it to a metric. Lock in a primary goal. And I would design your experiments very deliberately as well. Like, use our sample size calculator. Choose the right statistical model that's important to you. By the way, we are introducing additional models as well, like Bayesian as well. There's additional road map items for health checks as well during experiments that are coming out. Set up the feature flags and cohorts that you can really target the right set of users. I think Mixpanel gives you that. Standardize the analysis. Make sure the governance is set up early. All those kind of reset metrics as mentioned. Make sure it's all documented. But yeah. Okay. I think that was everything that I that I wanted to cover. We do have a little bit of time left. So yeah. I just want to review the chat here and see if there's any additional questions. What's the sample size limitations in Mixpanel? Do we use large number of users? I I don't believe there is sample size limitations. Let me double check that for you. But I do I don't believe there's any major sample size limitations. We are running user clinics over the next couple of weeks as well, So that's something for us to to check-in on that one. Yeah. Sure. So that is something that, yeah, we will we will take offline. So I'm just reviewing the q and a section here. Don't know. Amit, was there any other questions that we missed? Yeah. Sure. I've I've there's a question here at the bottom. Sorry. I've just seen in relation to, like, technically how the how the the how the variants dictate what happens in the client. Yeah. So just to just to recap effectively, it calls Mixpanel's feature flags API. So when your when your flag calls our feature flag API, we evaluate the flag in real time. We use the user identity. We use the context. We use the rollout logic that you've applied when you're determining the experiment, and then it will return the the consistent variant for the user. So we'll do the kind of evaluation in the API to decide or to sit to see whether that user qualifies for which cohort that you define for the experiment. If they do, then it will then it will return a winning variant for that or a specific variant for that user. So the mental model is the UI defines the rules, but then the code is applied in the feature flag within the code of your application, and that's what asks for and applies the decision. So yeah, you can safely control the rollout and experiments in the UI. Yeah. No no worries at all. I know we've got ten minutes left, everyone, but I wanna be respectful of everyone's time. I really hope you found that useful today. Bear with me. I'll just share my screen one last time just to wrap things up here. And let's share my screen. There we go. K. So we can see that. Yeah. Great. So we've done q and a. Tell us how we did. My call to action is please please scan this QR code. I would really appreciate any feedback, good or bad. It'd be very helpful for us to ensure that we know how to improve. But also, at the same time, if you're if you're a Mixpanel customer, reach out to your account manager. And my colleague will will pop a link in the chat as well now. If you're not a Mixpanel customer, if you wanna ask us any more questions or have further details about experiments or just overall Mixpanel, there you go. Larissa just popped in the chat now. Feel free to use that link and contact us there. But really appreciate everyone's time today. Thank you so much. I know everyone's super busy as well coming back after the the holiday season. So really hope you found that useful. And I really look forward to seeing you all again sometime soon. Thank you. Take care.