230: The Data-Driven Way to Make Decisions (Parenting, Health, Career): Emily Oster

That's Emily Oster, a Harvard trained economist who is the expert on making data-driven decisions when there's not always good data. And I'm a huge fan of her data-driven parenting books, and they've helped me raise my own kids and parent them in a way that I feel really comfortable. Today, she'll give us the key, the method to actually making good decisions even when there's poor data, there's not data, or we're in a lot of uncertainty and there's a lot of unknowns. By the end, you'll have a great framework for making great decisions like a data analyst, even if you're not one already. So let's go ahead and get into it Emily Oster is the founder and CEO of ParentData, a professor of economics at Brown, and a three-times New York Times bestselling author. Emily, welcome to the Data Career Podcast. Thank you for having me. Super excited to have you. I am a big fan. I have the, the books right here. If you guys haven't checked out- Amazing Emily's books before, definitely, um, check them out. They are amazing. Um, but we live in a really interesting time, Emily. Uh, there's, like, so much infor- misinformation going around, um, in parenting and in everything in politics and finance. Um, you know, everyone's trying to tell you how you should parent your kids and what decisions you should make for your kids. Uh, so my question t- for you, is it possible to make good life decisions in a world where we're constantly bombarded by different opinions and different data sets? I believe yes, uh, but I think it requires us to think about the structure of our decisions rather than just ask the question what the data says. So a lot of times people will come to me and they'll be like, "Okay, well, just tell me what the data says." It's like, that's not always that helpful a question, and if your approach to decision-making is to just, like, see the last piece of data and, like, make a decision based on that, you aren't necessarily gonna make good decisions, and I think part of what makes our current information environment so challenging is that people are constantly getting bombarded with data, and every time they see a new piece of data they're, like, not necessarily ready to incorporate it into their decisions in a smart way. So I think the answer is yes, we need data, and we can make good decisions, but we have to have the decision-making sort of lead the data. So I would tell people, like, you need to wait until you're ready to make a decision, and then think about what your choices are, structure the decision, and then you get the data that you need to make the decision and then make the decision based on that. But I think it's, it's too hard to Only use data, I guess if that makes sense. Yeah, for sure. I think, I think it's also interesting when we're talking about data to maybe specify what we're talking about. 'Cause, you know, some of the topics that, that you take on, um, like for instance Encryptshe, is, you know, is breast best? Like, is it actually good to... Is it better to breastfeed your, your baby, or is bottle feeding okay? Um, another, you know, one of the other things you tackle is vaccinations. Do vaccinations, you know, cause autism? And I think a lot of people, maybe for people who are listening to this, they're data nerds and, you know, they're able to, you know, maybe go out there and try to find some data on, you know, autism rates and vaccination rates, and maybe, you know, put together some sort of a statistical analysis to do so. But I think a lot of people are getting their data from, like, Instagram posts. Yeah. Um, so, like, how do you try to navigate the, the world where it's, like, a lot, where a lot of data is presented to us in, like, an Instagram post or something that's, that's maybe not very structural and, and hard to interpret in the moment? Yeah, I hate data from Instagram posts because it's always like, "Here's a study that shows blah, blah, blah, blah." And it's like, well, what, like, is it the only study of this topic? Is it the biggest study of the topic? Is it the best study of this topic? Is it some random thing from 1987 that you pulled out of, like, the journal of, like, made-up results? Which is usually the answer. And so i- Er, data is it's like I get so frustrated because I think we, we really need to prioritize the best data, but part of what is very challenging for people is it can be hard to know what that is. And there is a fair amount of training that goes into the question of like, is this good data? Is this less, less good data? Um, so I guess I would say it is never a good idea to make a decision about something based on a single Instagram post. If you are in a position to need to know whether some relationship is true, are vaccines causing autism, for example, you need to step way back out of Instagram or out of TikTok or whatever it is and figure out what are some sources you can go to that are gonna give you a better, more nuanced, more thoughtful answer to that question. And there are a few things people can look for in, you know, what makes a good data set, things like, is it big? Is it likely to be randomized? You know, things like that, and that's, that's kind of the core. But, you know, a single study says and somebody puts it in a carousel on Instagram, that's a crappy way to learn about data. That- that's... I mean, that's unfortunate. I wish we could always just like trust what we, what we see online. Um, but obviously we, we can't. Cannot. That's one of the things that I think, um, you do a really good job in, in Crib Sheet especially of like, you know, we're, we're debating, we're debating something like, uh, should we like co-sleep with our babies or should we sleep train our babies? Um, you know, you pull up like all these different studies that have been done on that. And some of the studies like that, maybe let's just say for example, that say, oh, you know, you know, co-sleeping is like really good for your baby actually. Um, you might throw... Correct me if I'm wrong, but like you might like throw that study out the window and kind of ignore the results because maybe it's not a large sample size or maybe it's not a diverse sample size or, or maybe like the actual experiment was wrong. So even though the results say something, like you're not necessarily one to just trust, uh, the results kind of randomly from a, uh, an experiment. Is that kind of correct? Is that kind of your way of thinking? Yeah, definitely. I think that a lot of what distinguishes the way that I approach sort of large corpuses of data from the way that you would and it sort of, um, that other people would perhaps, is that I am much more willing to say, okay, let me find the best studies here and base our conclusions on the best studies rather than just like every study should get their, their voice. Like some studies don't deserve a voice. Um, and I think that is especially true when we're outside of the randomization space. So in the parenting, like health, et cetera space, there is a huge amount of what we're told that is like we're just comparing people who do one thing to people who do another thing, and those people are really different on like a billion dimensions, and we're attributing it to the one topic that we're studying. And that kind of evidence I will almost always say like, just forget it Like just put it in the trash. And that is actually a place where I really differ from a lot of even, you know, people who I think have a lot of training and are in, you know, serious like professors because there are people who will tell you, "Well, okay, but we have so many studies of something. Like there are so many observations." But if it's not causal, it doesn't matter how many observations you have. And so I'm, I'm very interested in how we can have good data, really excellent data that lets us make causal statements. And if we have a study that isn't gonna let us make causal statements about something we wanna make causal statements about, I just think we should throw it in the trash. That's all. A- a- and, and I think that's important for people to realize because, um, I think there's some group out, out there who don't really take any scientific studies and, like, white papers. Like, they don't ever look at that. Like, they're only in the Instagram world, you know? Uh, maybe, maybe looking at aggregations of things. And there's some people who, who maybe are a lot more, like, prone to, like, "Oh, I really trust science, um, a- and studies," but it's always important to look into those studies. Um, I'm curious, like, there's not always a study for, for everything. Right. Um, so, like, what do we do when we need to make an im- an important decision, we wanna be data-driven in our approach, but, like, there isn't good data or there's no data? What do we do then? Yeah. So I think the f- first, that's very hard, and we have to first... I think first it's like there's a radical acceptance of just saying, like, "Hey, I'm gonna have to make a choice here. There is no option to, like, wait until the data is better. There just, I have to move forward with one thing." And if we paralyze ourselves w- with the view that, like, we can't make decisions until there are better data, like, the decision will be made for you in some direction by you waiting. So just to recognize, like, sometimes you'll have to make decisions under uncertainty, and that is unfortunate, but it is the way it is. I think the second thing I would tell people is in almost all of those settings, it's not important, right? So if something were really... It's not uniformly true, but if something is really important, and I talk a lot about parenting, but, like, really important in parenting, like really, really matters, you will see it in the data. Like, the things that we know really matter, like poverty, whether your kid has a stable place to sleep, whether they have enough to eat, those things really show up in the data. The correlations are really, really big. We have good causal evidence. The kinds of questions where people say, "Oh, I wish I had better data on this. You know, is it better to enroll my kid in travel soccer or in, you know, travel lacrosse?" Or where, like, there's no data on that, but you know what? It's not important. And so I think just, like, dialing down and asking ourselves, "How likely is it that this thing matters? Given everything else about my family, like, is this likely to be an important decision for my kids' outcomes?" And most of the time you're gonna say no. And then that actually makes the decision-making much easier because it allows you to focus on all the things that do matter for good decision-making, like how will this logistically affect my family? How much does it cost? Whatever are the things that really should go into that decision. So I think just reminding yourself you gotta make decisions when it's uncertain, and a lot of things are not important. And it's okay to say, like, "This probably isn't important. Maybe it matters a little bit in one direction or another, but on the whole, it's not the thing that's gonna break or break my kid," which almost there is nothing like that. I think that's such an interesting approach, and, uh, I'm just thinking, you know, about the people who are listening. Uh, like so many of us are data engineers, data analysts, data scientists, and it's like our whole life is, you know, helping businesses make better decisions with, with the data. Um, and so I think it's hard for me, like as a data nerd, to be like, oh, you know, sometimes the, the data doesn't matter. Like, or, or even maybe, maybe the choice, um, doesn't really matter. Um, but, but we're- I, I would... So let me tell you, I think this actually, there's such a strong, uh, like data analyst parallel here that I would make for people, which is like, you know, the difference between like great success and not great success in your business is like did you launch the right product? You know, did you... Like there's some big strategic decision that is happening, you know, usually above the heads of everyone and like where somebody at the top is making a big strategic play in one direction or another, and that's gonna determine like whether the business is successful or not successful. The job of the data analyst, and I do this like for my own business, is to be like in the weeds and be like can I get 1% more if I like send the email in this way or this other way? Or like can I optimize the pricing in this way? Like let me do an AB test on this, that, and the other thing. And like those things are really important for your business, but they're not important like the big strategic question. In the background, they're kind of optimizations on the margin. These kind of choices that we sometimes get obsessed with with our kid, they're optimizations on the margin, and that margin for parenting is really small, and there's so much noise And so thinking about like, if only I could optimize this tiny thing, it's like, well actually that's not important. Like, that's one tiny AB test in one place and like if you get it right, you, you don't get it right in parenting, it's, it's a lotta noise. So I don't know. That's, that's often how I think about the parenting piece. I mean, you bring up a good point because it's like if, should we launch product A or product B, and let's say we choose, you know, product A, but actually product B was the right choice, but there was just not data to make that decision. You know, what we're optimizing, let's just say like a subject line on an email, and you know, you AB test it and you find, oh, we get a lot more conversions with, you know, this, this subject line. It's like well, the bigger decision was really we shoulda gone product B. Like, the, the amount of money you can make on the- Totally you know, having the right subject line really is probably dwarfed by actually launching the right product. Which that's, that's an important thing to realize. I think it's important for people in, in, you know, in their p- careers to realize that as well. That it's like we're, we can analyze data, um, but we're, we're always trying to do so for business purposes and, you know, m- move the business forward. So it, whether that's for our, our kids... And, and even like you said, like travel lacrosse versus travel soccer. Let's say that there was data on that. It's like, well who's to say that your kid is like the average- Totally you know, it, it- Soccer kid doesn't take into account. Yeah. Yeah. Yeah. It's like- No, it- you don't have data on your kid. Totally. And I think in our, in our parenting decisions, of course like everything in our family is so much more complicated than like the subject line of the email that you're optimizing. And so every one of these decisions has some other decisions associated with it which may actually be more important. So we can get very like laser focused with our kids on thinking, you know, well let me... What is the right activity or choice or school to like optimize, you know, some outcome metric that we have attached to our kid, without stepping back and saying, you know, really what we're trying to optimize is like that our kid is a, you know, happy, productive adult who likes us and comes home for meals. And like that's actually a much broader optimization than like, you know, are we gonna Achieve Junior Olympic status or whatever, which you won't. One of the things I, I really like, uh, along these lines and in the, in the book is, you know, you mentioned that parenting you're gonna have a million decisions, and there's no way to guarantee that you're gonna make the right decision. In fact, you're probably gonna make the wrong decision quite often. Um, and instead of optimizing for making correct decisions, you talk about, um, making decisions through, like a framework and, and having education around the decision that you are making. Can you walk wa- about the difference between, like, making right decisions and making the decision the right way? Yeah, absolutely. So, think the most important distinction is one of those things you can do, and the other one you cannot. So you can never guarantee that you will make the right decision. It's just, like, not, that's not available to us. But we can say ex ante that we approach the decision the right way, and so I talk about, you know, being structured in how we make our choices, and starting by really outlining what our choices are. So being clear on the choice that you're making. People often will ask me, you know, "Well, should I do this or not?" You know, "Should I send my kid to this school or not?" It's like, well, or not is not an available schooling option, so, like, you better tell me what's on the other side of that, because you're never gonna be able to compare something to, like, the vast array of other things on the planet Earth. So you gotta think about what your choice is. You've gotta collect the information that you need. You need to then actually force yourself to make a decision, and I think that's the hardest part, because in a world in which we want to make the right decision, we can get, like, paralyzed by the realization that we cannot guarantee that. And so in order to work ourselves past that, we often have to really put in place, like, okay, I am gonna sit down and make a decision. Like, I, this is the date. I'm putting it in my calendar. I'm scheduling it. Like, this is the time we're gonna make whatever is this choice. Because otherwise you can just, like, spiral and spiral and spiral forever and never actually make any choices, and then usually the world will choose for you in some, in some way if you don't do anything. Um, and I, I think the, the value of having this kind of structure to a decision process is that on the other end of it, you can be confident, again, not that you made the right choice, but that you made the choice the right way. I think that's quite protective for people. I will say one other thing, which is I tell people, after you have made this choice, you should make a plan to revisit it. You know, there are some choices we can never revisit, you know? W- should I or should I not have a second child? We- well, once you choose that, you've pretty much, that's, you're pretty much committed. We're not revisiting that. But many of the choices we make you can revisit, right? Like, I sent my kid to this school, but I can choose another school. I sent them to this activity, but I could choose another activity, like, later, or we could not do this activity. And I think we owe it to ourselves to plan this kind of revisiting of the choices that we make, because if we don't, then we will never revisit them. And if we plan to revisit them, we're much more likely to do so. So much good to unpack there. Um, one, one thing I wanna zero in on is, like, you, you mentioned if you're deciding between A and B, and you're like, "I don't have enough data to do that," um, sometimes we don't make the choice, and that itself is a choice, is what it sounded like you were saying. And oftentimes it's w- the worst choice of the three, it seems like. Like, it's even worse than deciding wrong. Yeah, I think it, it sort of like, it, it, it robs you of the opportunity to improve the choices you have, right? So this is like, decisions are particularly hard if neither choice is really something we want. If we're sort of like, "Well, both of these choices are kind of poor," we're, we're really reluctant to make them because there's something very aversive about choosing something you don't want. But if you don't on purpose choose, you will end up in one of the, in one of the s- the branches, typically. And if you haven't chosen it, you won't have had an opportunity to make it as good as possible. Like, among the bad options, how can I make this option the least bad? And that's, uh, that's missed out on if you just decide to ignore the problem and forget about it. Uh, that's a good lesson for me to learn, because I am the king of doing that, and, uh, that stings a lot of the time. So, uh- I'm gonna try to do better with that. Um, I'm curious, like, so I'm used to data, quote, unquote, as, like, a table in Excel or, like, in, you know, in a SQL database type of a thing. Um, but like you said, we don't always have data to, to make decisions. One of the things that you talk about in your other book, The Family Firm, is, like, other ways that you could potentially get data that aren't necessarily from, uh, like a, a spreadsheet. And, you know, maybe they're not as high integrity as, as spreadsheet data. Um, but they're still valuable in making decisions. Um, so, like, one of the things you mention about is, you know, information about your personal circumstances. And, and you mentioned earlier, like, you know, if you're making decision A or B, like, which one are you actually going to like more? Even, even if one's more optimal than the other, like, which one are you going to like? And the other thing that you mentioned about is talking to, to others. I just am curious to hear your thoughts on, like, when you're getting data that isn't necessarily, like, tabular data, really high quality data, um, it's, it's more, like, around you data, what, what are you looking for? What signals are you looking for, and how can you know if you can trust it? Yeah, so I think really here I'm talking about getting i- e- information, I think. Look, data is just pieces of information. So I'm really telling you, like get some data on how you feel about stuff. And I think for many people who are kind of like us, actually maybe you wanna put that in a spreadsheet. Like I'm not averse to the idea that like you should collect data on your preferences and logistics and constraints in the same way that you would imagine collecting data on, you know, k- test score outcomes or, or whatever it is. But I think a big piece of this is, is kind of th- interrogating, like if I make this choice, what, what are the actual implications? You know, both how much am I gonna like my day-to-day? Really think about it. Like if I get up and I face, you know Le- let me put a concrete example in it. So a lot of people talk, talk to me about like choosing between preschools. Like I have this preschool and it's right close to my house, but it, you know, isn't very fancy. N- only half the teachers have master's degrees or whatever, or there's this preschool that's like 40 minutes away and it's like super fancy, right? It's like how do I think about that? Okay, and so one piece of that data is, you know, how much do we know about differences in preschool outcomes or whatever. And another piece of that data is like how do I feel about commuting 40 minutes each way with my kid? You know, what's that gonna do to my day? Like what's my day That's a piece of data. Like what is my day gonna look like? How are we gonna manage this logistically? How am I gonna feel about, am I gonna fight with my partner about this? Like are we gonna argue every day about who is gonna take their kids? That's a piece of data. You know, so there's a bunch of stuff in there that you want to put in your decision making, even if it isn't numbers in a spreadsheet, but it is information in a document. Uh, and another piece of that is asking other people, you know, what do they think about it? How did they experience this? But like- It's... For people who love data and evidence, there's an aversion to the idea of, like, going with your gut, and people will talk about this as like those are two choices. You could go with the data, or you could go with your gut. My view is like the data's not bossy. It's not gonna tell you what to do. It's only an input to decision-making, which also needs your preferences. People say, "I'm going with my gut." What they really mean is, "This is the thing I want." Okay. But you can actually bring those things together, and that's gonna be better than your data or your gut alone. I'm, I'm laughing because I'm literally... Like, we're putting, uh, our, our daughter in preschool. Literally went through that exact... Do we go to the s- the preschool we think is, like, better, quote unquote, you know, but it's further away, or do we just kinda do one that, that's close by and near? So, uh, I'm laughing 'cause I went through that, that exact analysis- Yeah recently. I will tell you, we went to the, we... The better one that's further away. It's not 40 minutes away. If it was 40 minutes away, I don't think I would be able to do that. But now that we're in the process, and, like, my wife is the one who takes her the majority of the time, it's like, "Oh, we don't really like this commute.” So, you know, next year or next kid or next semester, I don't know if we're gonna do this or not. Um, so sometimes you make a decision the best you can, and you make- Yeah maybe a wrong decision, but it's just another data point where it's like, "Oh, actually, I don't like being in the car for-" Yeah X amount of minutes a day" We learned this- And we're done totally, and then I think that's why you need the, the other step because if you don't plan to revisit that choice, then you, you're not gonna do it 'cause you're not gonna wanna admit that you made a mistake. And it's like it's not that you made a mistake. You did a trial, and, like, maybe that trial turns out like, well, we learned something that we wanna do something differently next time. A- and I think it's important that not only we do this with, with preschool, but when I worked for ExxonMobil, I was a data scientist there, and one of my jobs was to make machine learning algorithms to predict how much gasoline we should buy in each, you know, one of the thousands of different stores we have across America. And I, I built, you know, a machine learning algorithm that, that was the most accurate we could make it, um, to predict, you know, gasoline. But it wasn't like, hey, the, the gasoline prediction that my machine learning model puts out is what we order. That, that number goes to, uh, a trader, a buyer of gasoline, and they can totally ignore my number, and they can, you know, use my number as, as aid. So it's so interesting that, like, you know, these preschool decisions are kind of the same framework that, you know, multi-million, billion, you know, in- industry decisions are making. I'm sure you've seen that kind of with, with your research and, and, and your analysis as well. Totally. And I think we under even... It's, it's sort of interesting always for me to watch people who are so good at this approach to their job, who are so, like... And then I'm like, "Well, can't you..." Like, you should just be porting that into your life. Like, it's the same thing. Like whe- you know, when you are running, when you are married to someone and you have, or you like have a partner and you have kids together and you have two jobs and whatever, like you're running a small to medium sized enterprise. And there actually are like quite a lot of tools from how companies do that, that I think will make this easier for, for people, even though it's like a really weird thing to say. And when you tell people that they're like, you know, they're like, "Well, do you boss your husband around?" It's like, "Of course I boss him around," but that's not because of the, it's not because of this enterprise idea. That's awesome. And I love that. And that's, you know, one of the, the things you talk about in, in The Family Firm is like, we should, we should really kind of organize our family, uh, like we r- organize teams and, and businesses in a fun way, not like in a boring way. Yeah. But like- Fun way we should have objectives. We should have goals. Like, we should, you know, make decisions that are for the optimal happiness and health o- of our family, which I think I've, I've been reading it. I've really been, um, enjoying that. Um, I'm curious, so you know, we talked about like making data-driven decisions as, as parents. I'm curious, like where else in our personal lives, um, that we can make data-driven decisions and kind of adopt the approach that you've taken in, you know, in breastfeeding and in, you know, sh- what kids, where, where should we send our kids to school? What other places in our life can we live data-drivenly other than parenting? Yeah, I mean, I think the other obvious one that people like is health, um, is sort of like there are a lot of health decisions that you have to make. Uh, and there's a lot of data on those. Uh, and I think it has many of the same issues that parenting has. You know, there's some of the data is better than others. Um, and You know, we, you have to make decisions that take into account your constraints. Um, and I actually think this is a place for me where we are... A mu- much of the discourse misses the idea of constraints. We're like really good in the health space about talking about like how to optimize, like, you know, let's track every da, da, da, da, da. Like, you know, how do you like get all of your n- numbers to be exactly optimal in these various ways? But without helping people sort of see like, okay, well, you probably don't have 17 hours a day to like fully optimize a 27-step life protocol. Uh, and so how do we incorporate the data with the constraints and ask, you know, what are, what are the most effective things, uh, to do? And I think the other place is just how, in like sort of general, like how do we choose our jobs and how do we, you know, operate our like professional lives? Um, but I think health is the, health is the other obvious space for me. Health is a really interesting one, um, because obviously, like, it- it's If you don't have health, you have nothing in your life, right? Right. 'Cause, like, I think we all have known someone that's, that's lost health, and we just see, you know, how much of a detriment to life that is and how, how difficult things are. So that makes sense to optimize it. Um, just, like, a, a concrete example of that is, and I know you, you talk about, um, this in, in your books. Um, but I got diagnosed with ADHD last year, and I had a, I had a decision. It's like, do I try, you know, after trying, you know, six months of non-medication ways to, to, to solve the problem, like, do I try medication or not? Yeah. Um, and you know, one of the things that I do is, like, I pretty much constantly wear, uh, an Apple Watch. And so I've been taking, you know, ADHD medicine for, for almost nine months now. And one thing I've seen is my resting heart rate has, has risen, like, five beats per minute. Yeah. And it's like, okay. Um, that's, like, a side effect of taking ADHD medicine. It's like, do I, do I like that? Do Is, is the health risk that with my heart worth the effects of m- you know, maybe me getting more work done or maybe me being a more patient parent? Um, those types of things. And it's hard because it's like there's not really all that data out there that can support Or, or maybe there is a bunch of data out there, but it's like which one do I trust, and how do I apply it to my personal life and my business and my family life? Um, so I guess making decisions under uncertainty with health definitely makes- Yeah makes a lot of sense. Yeah, and un- under uncertainty, and they're really also under constraints. Like, you're, like, really what you're describing is a constraint, which is like you, like, if you do this one thing, it has this effect. But y- like, you're It's trade-off, uh, and we're not that good at trade-offs. And a lot of the health messaging in particular sort of doesn't like to acknowledge the existence of trade-offs, so they're just like, "Do all 4,000 of these things." And it's like, okay, but I, I, I can't do that, or it's, like, literally impossible to do two of these things at the same time, and so now I have to pick one. And, you know, people ask me, like, "Should I sleep or exercise?" Like, I only have 30 minutes. I have to pick sleep or exercise. Like, which thing is better? And, uh, that's, that's a hard question. What's the answer? Hmm, kind of, kind of depends how much sleep you're getting, but probably sleep. Yeah. I, and ob- obviously there's so many factors that, you know, that go into your life, and it's hard to, to make a blanket statement. It's funny that you're, you're mentioning this, 'cause my other job at ExxonMobil, my other problem that I solved at ExxonMobil, uh, and I didn't solve this problem on my own. We worked as a really big team to do this. But we, we made mathematical models of the entire refinery, um, which was, like, 140,000 equations. Um, and we were trying to optimize, you know, how much money the refinery can make with all these different constraints. Like, we can't... We need to make sure that our, our pollution's abov- uh, below this level. You know, our, our tower can only take this many- Yeah barrels of crude every day. And that was a really hard problem to solve. Yeah. Um, and we knew, we knew all the, the math behind it. It's like you can't really do that in your life, 'cause, like, you can't really model life outcomes, I don't think. Nope. Um, maybe you can. I don't know. No, and you have... And, and of course, then when people try to, they come up with, like, crazy things. Somebody sent me a paper the other day which, in which these people, like, tried to assign a number of minutes of life to, like, each food. So, like, if you have one Diet Coke, it costs you, like, this many minutes of life. But it's like, that's a cra- like, first of all, that's bananas. Like, you definitely can't do that. It's all of the data is from correlation. It's not causal, whatever. But it also just, like, didn't make any sense. It was like a Diet Coke costs you 12 minutes, but, like, a peanut butter sandwich gains you, like, 33 minutes. And it's like, okay, well, if I eat them together, can I get fif- like how does this work? But it was so, like, so much in the space of people just want an answer. They wanna know, like, okay, how much is... Like, what's the cost of this Diet Coke? And the answer is, like- We don't, you know, we don't have it, probably zero. Uh, or have it with a peanut butter sandwich, and then I get to negative 17. That's awesome. I love that. That's, that's very cool. Um, okay. I saw something really cool that you, um, posted on your Twitter recently and your Substack, and we'll make sure to have a link to your social in the description down below. Um, but it was a really cool analysis you've done recently on New York education data and kind of like their testing data. Um, and one of the things you actually published that caught my eye was a Claude artifact. Yeah. So for those who are unfamiliar with Claude, it's basically like ChatGPT, but it's from a different company called Anthropic. Um, I really like it for doing things like data analysis, and it creates these things called artifacts, which are basically, you can think as like a, a, a published something, a URL that goes to some sort of a page that has text on it. And in your case, you were analyzing data, so it had text and graphs and different analyses and like that. Um, and I wanna talk about the New York- Yeah education study, uh, study that you did. But first, I want to kind of walk me through your, like, data pipeline. Like, how, how did this, like, come to be? Like, where are you getting your data? How are you analyzing it? How are you publishing it? I was really curious about that, if you don't mind sharing maybe, like, a high, high, uh, view of that. Sure, yeah. So in, in that case, actually, the key to that entire analysis is one of the projects I d- I do is something called the Education Data Center, uh, which is a, a project where we, uh, try to clean and organize all the state-level test score data. So if your kids are in, you know, public school in grades three through eight in the US, they will take, uh, math and ELA tests every year. That's like an important part of accountability. Uh, but the state's data is, like, a hot mess. Like, every state is issuing it in a different thing. If you wanna have the data from Montana, it's 3,000 separate spreadsheets. Like blah, blah, blah, blah. And so one of, one thing I really care about data transparency. Uh, and so in this project we, like, download all of this stuff, and we have, so we have like a, a website where you can sort of get all the microdata for, for this. And I mention that because that's a sort of core, like, backend pipeline for a project like this. On that particular project, there's a, this thing that happened in New York with the test scores, which we can talk more about, but where, like, basically somebody called me, some reporter called, and they were like, "Here are the, you know, here are the test scores that are gonna come out. Like, what do you think?" And I looked at them, and I was just like, "They're wrong. Like, I don't like, I don't know what to tell you, but, like, data doesn't look like that. Like, somebody made a mistake probably last year." And then I got really, like- exercise. I, like, I really love the, the piece of data where you try to, like, learn what's going on. Just like, I just wanted to know what's going on. Like, I couldn't, like, let it go. And so then my data pipeline is, you know, in the, in the back end, I'm basically using Claude with an API pulling down this, this raw data and kind of writing code in Python to, like, figure out, try to figure out what's going on. And this is a place where the, these AI tools and sort of Claude in particular has really changed how quickly I could do something like this because, you know, on the back end, I could have written this code in Stata on my own and so on, but I probably would not have had the bandwidth to do it without a kind of LLM tool. Uh. It was awesome, first off. Uh, it was super cool. It was so fun. It was like, it was so cool. Yeah, so basically what, what I'm hearing is like you, you obviously know how to do this analysis. You could obviously do it from scratch. And I love the idea of from scratch. It's like none of us are actually doing this by hand and paper. Right. Like, that's probably from scratch. No. So it's like, oh, like- My dad was an economist. He used to like, I think, do this by... They would like multiply the matrices, but that was a while ago. See, but that, that's kind of my point, is it's like, oh, and then R came out, and then Python, or and then, then Stata- Right or whatever. Uh, you know, and it's like now we just have Claude and ChatGPT, which I just see as like a new tool, like you said, that enables this- Yeah type of analysis. Um, and it's like I don't think we could've taken a random Joe off the street and, you know, had them create this analysis that you created. I don't think this analysis, the AI could've created on its own. Um, so it's cool to get like a little bit of glimpse on, on how you're using AI to do that. Um, so that, that's very cool. Um, I do wanna get into like the, the details of, of, of what happened in this, in this, I wanna call it a study, but it's not a study. These test results. So basically, um, if I'm understanding, uh, correctly, the test results... L- l- let's make it as simple as possible. The test results were around a certain level, and then the next year they jumped up like 10%, from like 43 to like 52%. And maybe we'll pop up the, the graph on the screen that your Claude created to, to show people. Um, and then they've fallen back down to normal levels- Yeah this year. Yeah. And so what you're arguing, if I'm not mistaken, is basically something happened in that middle year where it's like, no, we didn't see improvements of 9%. Like, something weird happened. Like, there was some error in the testing or some error in the analysis- Yeah where it's like... And this is important because it looks like the state's doing a great job, we're really improving, when in reality it didn't improve at all, and that's, that's really big implications on like funding and money. Is that correct? Yeah, absolutely. I think the, so, so two things I would add to that. So one is it, it really can't be that they s- the scores went up this much and down this much. Like, this is a place where we have so much data on how much test scores like this vary. We know so much about just what is going on. And, and the numbers here would imply that like the tip, the, the sort of across the entire state in basically every school across every demographic group, like fourth graders in one year learned two-thirds of a year more, and then in the next year they lost all of that and more. Like, it's just like this isn't, you know, it's, it... No. This is not right. And I think that's a piece where probably the person part of this is really like I, like I have so much experience with this, I can just look at that and be like, "That's wrong." And now I can go into, you know, some LLM and be like, "Okay, I'm sure this, like I'm pretty sure this is wrong. Like, let's try to, try to understand it." Um, and okay, so that's the first piece. And then, yeah, the question is what, uh, like what, what happened behind this? And it really does matter because as you say, you know, funding decisions are made on these, on these numbers. And for example, New York allocated, you know, some three-year funding grants of $250,000 a year across schools- based on these flawed test scores, which like basically made more middle schools get this and some elementary schools not get these like large grants. There's like a lot of money behind, millions and millions of dollars behind these scores, and some of them are, in my view, wrong. That's, uh, crazy, and thank you for, um, you know, organizing this and trying to suss out these things. That's, that's really important, I think. Yeah. I w- I mean, look, it was... It- I think it is important, but it was also very interesting and fun because it required really, like, getting into, you know, like, well, what... Like, what did you do wrong? Like, what exactly? And that's, and that's where I think the, the ability to move quickly with these LLMs and the ability to have an LLM read, like, you know, like, 600 pages of technical documentation and be like, "Okay, you know, here, like, let's kind of problem solve. Like, where could possibly this have, have fallen apart?" And I got much further than I think I would have been able to alone. I still need... I, I'm not done. I mean, I'm, I'm done, but I'm... We're not done. But I think that somebody is gonna figure out what actually ha- happened, and hopefully fix it. Super cool. I hope. We'll, we'll include a link to your, um, Substack, 'cause I know you have, like, a whole Substack dedicated to the education stuff- Yes um, and that analysis as well. And I actually wanna talk about the education data, 'cause one of the things I do is I run a, I run a data boot camp, um, where I try to help people learn how to become data analysts. And one of the projects we do, the second project that we do, 'cause I'm really, like, hands-on, project-based, is we actually analyze, uh, the data from Massachusetts and the- Nice the, the results that they get from that. And so, one, I'm familiar with how messy and how many Excel spreadsheets and how- Yeah how hard it is to, like, join that data and- Massachusetts is actually very good relative to the average state. It's not bad. One of the reasons I chose it, 'cause it's the second project. We don't wanna get too hard and, like, actually joining and bajillion different things. Um, but first off, I was, like, super stoked to see, like, oh, lookit, this is, like, a really viable project that people are doing similar things in real life- Yeah 'cause we try to create a dashboard based off of, like, what's happening in the schools and who's doing well and who's not doing well. So I was stoked to see that, that. And the second thing, like, no, no pressure obviously, but one of the things we do is we have a team of data analysts on our, uh, you know, on our program who are always happy to analyze data, especially for good causes. So if there's ever, you know, another school that's cheating or doing something wrong and you don't have the bandwidth, we're happy to, to, to, to do an internship project and, and- Oh, that'd be awesome analyze that data and try to give you- All right our results. That would be very fun. Yeah. I think we- Yeah we are, in that project, we are so focused on the, like, just getting the data together, and I think that sometimes we... Like, there isn't bandwidth to do the, like, okay, can we really understand why these things changed in the way they, they did, other than mistakes. Okay. Well, I'm, I'm serious. Maybe we'll talk offline on, on how to do that. Yeah. 'Cause I have so many people that would voluntarily do some pretty interesting analysis. Um, okay, that was awesome. Okay. Um, the, the last thing I wanna do with you is play a game. And, uh, you know, you are the queen at, like, taking a complex problem and being like, "Oh, you know, this is what the data says. You know, this is, this is, like, whether the data's good or not, this is what maybe you should do in your life." And- Okay uh, you're, you're very good at nuance. But I wanna ask you a few rapid fire myths, and you tell me in one sentence, uh, whether it's true or not. Does that sound good? Okay. Yeah. Great. Okay. Number one, uh, is Tylenol safe in pregnancy? Yes. Tylenol is safe in pregnancy. Okay. Perfect. That's easy. Number two, is breast best? Breastfeeding has some early life benefits, but many of the benefits that you are sold on, like IQ and obesity and so on, are not supported in the best data. Okay. Number three, are phone bans in school a good idea? Yes, but not because they're going to dramatically change a lot of test scores, but because they are good for kids' interactions with each other. Awesome. Number four, do cell phones cause cancer? No. And you know how we would know? If brain cancer had gone up a lot over time instead of actually what has happened, which is that it's gone down. That's good news. All right. Uh, is red meat bad for your health? No. Do vaccines increase autism odds? No, they do not. Um, should you take creatine? Yes, if you are strength training. There is no point if you are a sedentary person. Okay. And last one, can you get Botox while pregnant? You can, but no one's gonna do it for you. Okay. There you go. Uh, well, if you guys want the more nuanced, data-driven, longer answers to all these questions, you'll find a bunch of in-depth articles, uh, on Emily's website, parentdata.org. It's actually one that I, I use pretty often when I have a question in my life, especially when parenting. Mm-hmm. Um, I subscribe to Emily's newsletter. We'll have links to those down below. I find them incredibly helpful, uh, with parenting. But even if you don't have kids, I think you'll find Emily's style of taking data and life information and making good decisions or at least having good frameworks for making, uh, good decisions really helpful. So we'll have a bunch of Emily's links in the description down below. And, and once again, check out Emily's books, The Family Firm and Crib Sheet. And there's another one that's Expecting Better, is that what it's called? Yeah, Expecting Better. Okay. That's the OG. It's about pregnancy. I, I was too late to have that one. I don't have that one, but, uh, maybe next kid, we'll, we'll get that one. Um- Next kid, next kid. E- Emily- Thanks thanks so much for coming on the Data Career Podcast. Thanks for having me