Transcript#

This transcript was generated automatically and may contain errors.

Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.

In this episode, we talk with Leilani Battle, who likes things she can repeat, study, and refine. Whether that means honing her disc golf skills, tweaking a recipe, or deriving the perfect vegan buttermilk sourdough pancakes. This same experimental instinct runs through her research at the University of Washington. Leilani built her career straddling two very different research cultures, the computation behind databases and the human-centered, experimental world of visualization, an experience she describes as essentially getting a PhD in two disciplines. We talk about the magic that happens when those fields meet, how what loads first in a visualization can shape what people find in their data, and how our work now applies that same technical and human perspective to responsible data science. So I'm so excited for you to listen to Leilani.

Leilani, welcome to The Test Set. So glad to have you. So Leilani Battle is the Robert E. Denning Endowed Associate Professor of Computer Science at University of Washington and co-director of the UW Interactive Data Lab, and also a member of the Database Group. And you do a lot of really interesting work. It seems like on the computational aspects of visualization, but also the cognitive human factor side of it. Why would we say speed up visualization and in what situations for people? And also more recently, it seems working a lot on responsible data science as well. So thanks. Thanks so much for coming on.

Thank you for the invitation.

Disc golf origin story

So often I feel like I'm like, at the very end, we get to people's hobbies. But I hope it's okay that we flip it around because I think you have so many incredible, interesting hobbies. I thought maybe we could go hobby forward, which was first and foremost, I do want to note, you mentioned you do a lot of pickling and sourdough. And I think it's incredibly important we get to that. But also, you said your most surprising hobby is probably disc golf. How did you, could you unpack for us your history with disc golf? Because I know you're in the Pacific Northwest. So I don't know if that's related, or how did you get into disc golf?

It's a good question. So long story short, my husband got me into disc golf, and he's been playing for a very long time, since high school, actually. But I didn't really have any interest in disc golf whatsoever until the pandemic, when we were all stuck inside, we couldn't necessarily go and do our more social hobbies. But I still wanted to kind of go out, hang out with my husband, just do something outside of my house at the time. And he had been suggesting disc golf to me for years. I think I tried it once in high school or college or something and was like, nope, no thanks. And then he managed to get me to do it again during COVID. Because he's like, we could go outside and be in nature and throw frisbees. And I was like, okay, cool. I'll do it again. And then that time something stuck, and I really fell in love with it. And I just kept doing it.

And something I think that helped at the time. So I was an assistant professor at the University of Maryland College Park. And we had bought a house that was like a 15-20 minute walk from a very beginner friendly disc golf course. So I developed this habit of every weekend, going with my husband to this course and playing disc golf. And I just kept doing it. And then when we moved back to Washington, Washington has so many beautiful disc golf courses, most of them free to play actually too. And there's a, so I never checked out the disc golf scene like the community in Maryland when I was there, but I started checking out here because I wanted to be able to play with more women. Most of the people I'd see on course are men. And I love playing with my husband, but I just wanted to kind of get into it more. And then I started playing tournaments in our local area. And that's really where I kind of shifted and really got obsessed with the sport.

Wait, I love that. I didn't realize you're, you're a competitor. You're entering the tournaments, disc golf tournaments.

Amateur, amateur division tournament. I could go on forever about disc golf. So there are mixed divisions, meaning you could be like any gender and play in those divisions. And then there are some gender and age protected divisions. So there are female specific amateur divisions and professional divisions, and then age protected divisions that start about every decade and eventually go about every five years as you get older.

Wes, have you disc golfed at all?

I played a little bit of ultimate Frisbee in high school, but it's been, it's been a long time. I'd have to, I'd have to really think hard. Like what's the last time I threw a Frisbee. It's been, it's been a while, but I mean, I think like, it's interesting, like all the, the changes to our hobbies and our personalities that were caused by, caused by COVID. Like, I think I understand like the people that got into pickling and sourdough and I didn't do sourdough, but I did pickling because all of a sudden I wasn't traveling like I used to before COVID. And it was like, what am I, I'm stuck at home. Like, what can I do that I didn't use to be able to do like, Oh, sit and watch a thing, you know, do science like inside my, my kitchen. Um, so, but you know, now I enjoy, you know, I haven't pickled lately, but it is, it is, you know, now like a new hobby that I indulge in from time to time.

Repeatable recipes and the science of cooking

And the, the pickling, I thought this was interesting in a profile that was written on you. They talk a lot about, a bit about you doing some life logging. I feel like you mentioned like, um, cooking in batches and enjoying sort of doing like big pickling batches. Um, I'm curious if you could say a bit more about some of your, yeah, why you pickle and what you like about it. Uh, I thought that was an interesting note. Some of the batch batch cooking. Yeah. Are these things connected?

So what I'll say is probably the most satisfying feeling for me when it comes to cooking and pickling any kind of food preparation is when I'm able to come up with a recipe or a combination that to me feels optimal. Like it, it's really easy to make. It tastes great and it's repeatable. Maybe it's a scientist to me. I love it when it's repeatable. If it's not repeatable to me, it's a terrible thing to cook because then it implies you can never do it again that well. So I think to me, these kinds of hobbies where even disc golf kind of falls into this, where I can just kind of do it over and over and over and hone the recipe, hone the skills. I, I don't know. I really like how it feels and pickling and cooking, even meal prep where I cook in bulk. Like they're the kinds of things where I can do them over and over and over and get better and better and better over time. And also where documenting, um, and recording my experiences or experiments, if you will, pays dividends.

If it's not repeatable to me, it's a terrible thing to cook because then it implies you can never do it again that well.

It sounds like you really like, you're solving the reproducibility food recipe crisis, but you like the fluency, like by, yeah, by repeating an action, you can kind of like see yourself getting more fluent at it from sort of like time to time.

Yeah. And I can experiment with different techniques. Um, I also like to do or call making base recipes where you can take kind of the core of that recipe and then modify it in different ways to get a, a different, like a fresh experience. Cause you can imagine if you ate the same food over and over, you might get bored. So having these kind of base recipes where you can mix things up, it's still repeatable, but gives you variety. Like I love things like that.

Yeah. Yeah. It's, it's sweet of you to think about the human factors too, that you're like, if I eat the same thing, I'll be bored with it.

That's just practical. I like to save money too. I can't, can't risk throwing all my food away.

I, I really love the, um, the ratio cookbook, I think by Michael Ruhlman, which is like, okay, actually across all of these different types of things, like mostly baked goods, there's just like the same underlying ratio, which I think is really cool. Like I loved, like I love learning like a pound cake. It's called a pound cake because it's like a pound of butter, a pound of sugar, a pound of flour, and a pound of eggs. And that like one, one, one, one ratio is like the basic ratio between basically every type of cake. Like I think that's just so like empowering and like cool to, cool to know.

That's exactly the kind of thing I love. Um, an example of a recipe that I worked on for a really long time and really enjoy doing, um, is like my buttermilk pancake recipe. So I started making buttermilk pancakes in graduate school and I got really good at this, but then I became a vegan, which kind of destroyed my buttermilk pancake recipe, at least the old version. And I had to rebuild it. But because I had all those experiences and what makes a good pancake, I figured out what makes a great vegan pancake as well.

Do you, do you, do you think we can get that recipe to share with our listeners, Leilani?

I don't know if it's secret. It's more like, I'm probably shy about sharing a recipe cause I'm like, Oh, what if other people don't like it? Or I don't know. It's like a side of my life I'm not used to sharing widely, but I am happy to write up the vegan buttermilk pancake recipe.

Yeah. Or just, or at the very least Hadley gets it and then it just shares the secret with you.

Yeah. I'll make it and then I'll, I'll write it.

Yeah. I should also mention it's a vegan buttermilk sourdough pancake recipe. So maybe not all your listeners can make it anyway.

I feel, I just, I just feel like there was, there'll be like, there's enough of our listeners who will be able to make it. Who are vegans, who have sourdough. I just feel like this is probably over-representative.

Yeah. If you have a sourdough stirrer, it's actually pretty easy. So happy to share.

Path to visualization research

I do feel like a lot of this does hit on, what seems so interesting about your work is a lot of that balance of like, there's a lot of computation in it, but also experimentation. Like how do we, for example, in your studies of visualization, like both how can we technically speed up visualization, but also like in what situations would a person benefit from this both like the computational problem of how, but also like how do we stay tapped into the sort of human centered part? Like what part of a person's task does this actually improve?

Yeah. I'm so curious to get into it. I wonder almost if just leading up to it, you could tell us a bit of how you got sort of to where you are studying these things. Tell us a little bit about your journey to where you are today.

I'll try to maybe sprint through kind of my path, career path. Initially, when I was in high school, I was really into drawing and arts and I loved playing video games. So I had assumed that because I was passionate about those things like drawing and video games or art and video games, that I would end up there somewhere. That being said, I think the influences of, you know, interacting with that kind of technology, you know, like game consoles, my family, I always had a computer at home growing up, things like that. I think I was more open to kind of being in tech in general. And thankfully I realized right before applying to college, like universities, that, you know, there's more to life than being a game developer or an artist or a designer for games. And I, like based on my dad's interests and experiences around electrical engineering, I figured, oh, that seems like a viable route if my dad was interested in it.

And this is kind of the first of some coincidences, I guess. But when I applied to the University of Washington, because I was, for me, Washington is home. I grew up in the state of Washington primarily, and the University of Washington would be considered kind of a home university for me. I knew they had a strong engineering program, including electrical engineering and computer science and engineering. I was directly admitted to the computer science and engineering program at UW because originally I wasn't even going to do computer science. I was going to do electrical engineering. And with that direct admission, I was like, oh, well, this is a pretty competitive program that's hard to get into. I can give it a try. And if I hate it, I could always do something else. And so that's kind of where I ended up in computer science specifically. So not every part of my life was, you know, carefully engineered to end up exactly where I am. There have been some kind of happy coincidences here and there. I think that is one of them.

So I started going through the program, started taking the intro, competing courses, ended up loving them, did not see that happening. But I thought they were incredibly fun. You know, kudos to back then the instructors teaching the intros courses, because I'm at UW now as a faculty member. So kind of seeing the other side of it, it's been really fun to see, you know, everything that goes into designing a really good CS course, especially at the introductory level and how that had such a huge impact on my life. So I was enjoying it. My early years as an undergraduate student, I actually struggled to get good, like competitive internship offers. And my first summer after my freshman year, I didn't have any opportunities to do a tech internship. So I decided to apply to research internship programs. And I did an on-campus research internship program that summer. And that was my first introduction to computer science research. It was actually in Professor Rajesh Rao's lab here in the Allen School. And at the time, he does a lot more like kind of neuroscience and engineering related things now. Back then, his lab was a bit more of like a traditional robotics lab. And I've mentioned this to him too. I'm really grateful for that experience, but I

knew I was not a roboticist. So after that, my next summer research experience was actually in databases, in the database group here with Magda Balazinska, who is our school's director. So she's, she's been in my life and in my career for a long time. And I really look up to her. But kind of seeing the database group, how it operates, the kinds of research projects they were working on, I started to develop an appreciation for databases. And the work that I, I contributed to in particular was a little bit more oriented in the human computer action direction. So thinking about how we can make people's lives easier when they're trying to use database systems. And I think that kind of set me up for graduate school.

So as I was finishing my bachelor's degree, I was looking at what opportunities I had. By then I was able to get, you know, more competitive internship opportunities and experience a little bit of what industry is like. And I just felt like I didn't have enough time to explore what I wanted to learn in academia before leaving and joining the workforce. I felt like I got a decent sense of what to expect in industry. I did not get a decent sense of what, you know, what potential I had in academia. So I wanted to keep going.

And so, were you pretty set on joining academia by, academia by that point or? I was somewhat conflicted because part of it was my parents not, so I'm the first person in my whole family to get a doctoral degree and including my extended family. Oh, that's incredible. Yeah. So for them, this was really a foreign concept and it's, even when I had made the decision, it took some convincing of my parents that I wasn't like throwing my life away by not automatically taking six figures from Microsoft or Google or whoever, Amazon. And they were really worried, like, are we going to have to pay for this? We already paid to help you go to college. Like, are we going to have to pay again? Like just a lot of kind of educating my whole family on what it means to go to grad school in computer science and what the benefits are and what the value could be long-term, stuff like that. But yeah, I was a little conflicted because it's a hard decision to make for a lot of people. It doesn't make sense to go to academia. It doesn't make sense to get a PhD. You can do awesome in industry without it. So, you know, it's a, it's a tough decision to make, but I, I landed on the side of, I want to do this. I want to try. I want to see what possibilities are out there.

The database group was incredibly helpful and supportive in helping me apply to graduate programs. And I ended up getting into MIT, which to me was really exciting. It also changed my parents' minds because they've heard of MIT. Yeah. That's incredible. Yeah. They're like, oh, MIT. Well, I guess this grad school thing would be okay. They're like, we could, we're, we're into it. We're willing to like, see where this goes. Yeah, exactly. It was at that point where they were like, okay. Some name brand school. I guess I get it now. I don't, no shade to my parents. I love them. It's just, they're kind of funny cause they don't really, they're not academics. So the way they see my job and academia is very different from the way I see my job in academia. But so then I went to grad school. I started working with Mike Stonebraker. He was my PhD advisor. I was in the MIT database group. Thankfully MIT also at the time and still does, it's even stronger now, but MIT had really great HCI researchers as well. And in fact, it was Rob Miller, who doesn't do any of his work, who connected me with another professor at Tufts University, Remco Chang, who is a visualization researcher. And it was with Remco and Mike that I started, you know, charting out what I actually wanted to do for my dissertation and learning what it takes to be a researcher, not only in databases, but also in visualization.

I did get a very small taste in conversations that people in the database group were having at the time at UW around kind of more HCI and visualization oriented work because there were starting to be some interesting intersections. Like I think at the time people were talking about like many eyes, which was like a pretty influential platform around back then when I was an undergrad and graduating and moving to MIT. But I didn't actually do any visualization research as an undergrad. And it was really in grad school when I started getting exposure to that and learning about it. It was a little tough to do it in a database group. I loved being in the database group and I feel like it was a strength as far as my technical ability and my strength as a researcher. I feel like I really needed that. But it's hard to say publish papers at visualization conferences when you're used to reading and learning about database papers and conferences. So I really had to find my way.

Databases and human-centered research

Like the database research community is like not by and large, like not that interested in kind of human factors, right? It's more about making stuff fast and efficient and correct, like not like easy to use for humans. Is that right? Or is it like, is that kind of like the HCI bit of databases, like a fringe community? Or is that kind of fairly central?

I love that question, actually, because there's some nuance to it. I would say a lot of people in the database community had thought that way when I started my PhD. And I think today there's much more of an appreciation for user centered database work. And I see more of it getting published at SIGMOD now than before. But on the whole, I would say database researchers do not focus on it. I think they can, if you spend time talking to them about it, they can understand the value, but they just don't prioritize it. And of course, there's people, there's going to be people out there who are like, Oh, this is a waste of time, or this is not my job. I don't see why I should focus on this. And there'll be other people, you know, like my PhD advisor, who really see the value and opportunity there. And don't see it as a sacrifice as far until actually speaking for database research.

I mean, I think what's been really interesting in the last, you know, in the last 20 years has been like the, that all the work that's happened at the intersection of database systems and data visualization and human computer interaction. And I think like the HCI community has, you know, embraced work happening in databases and data processing, because without like that fast slicing and dicing, like the cross filtering, you know, being able to like, think everyone when you see a really, like a really compelling dashboard with like the instantaneous interactions, like low, you know, having low latency interactivity, like being able to like drag a slider and see, like, sort of start to experience the data through interacting with an interactive data visualization. And then seeing like, Oh, the thing that made this possible was, you know, DuckDB, or the thing that made this possible was like this analytic database. And so like, without the database technology, you wouldn't be able to create like these compelling, you know, data experiences. And so I feel like there's this virtuous cycle between, like between the worlds. And so I'm, you know, as somebody who's like, deep inside that world, I mean, it's just, it must be incredibly, you know, incredibly interesting to be able to like, have influence, like on both sides of like, you know, sort of more of the HCI data visualization research, as well as like the computational side of like, how to like, what can the computations make possible that wouldn't be possible without it?

Absolutely. I think this is one of the most fun parts of my job. Because I love straddling that boundary. It was really difficult in the beginning, just kind of finding my way because you have to, I, I'm fond of saying I basically had to get a PhD in two disciplines to be able to talk to them. But now that I'm there, I really love talking about those research problems and understanding the strengths and weaknesses of both communities. Because when you put them together, they make magic happen. And, and one classic example that I love to point to that is kind of very easy, a very easy win for that intersection is, you know, Tableau and originally the Stanford Project Polaris. That was an example of how putting those two things together can make magic happen in a way that no one community could have possibly done that on their own.

Because when you put them together, they make magic happen.

Tableau and the Polaris paper

With Tableau, just to kind of unpack it a bit, are you saying so for context, for people to see maybe like Tableau is a, in a way a BI tool where a person can use like a graphical interface to click through their data and like chart it or present it as a dashboard. What, what, what are you saying Tableau brought specifically kind of to that, to the world?

Okay, cool. So what I love about this, because the Tableau paper, I think the original Tableau paper came out in like 2002. And at the time it was this research prototype called Polaris. And it was kind of funny too, when you read the paper, because the paper is, when you read the paper, the introduction is basically making the case for, hey, this community, this is why we should care about what databases do and how we can map our visualization primitives onto database primitives to automatically generate SQL queries to process our data. That's basically what that paper was bringing to the Viz community. It was saying, A, we should care about this. And B, our theory and database theory could actually be integrated to produce this really nice automated stack going top to bottom and bottom to top. And the grammar that kind of tied it all together is this, this grammar called VizQL. Basically this mapping across the stacks, because you can think of it as a stack, right? You have your data processing and storage layers, which the database management community typically owns. And then you have the visualization and interaction and user interface design layers, which the visualization community typically owns. And that paper was saying, why, you know, bisect the problem in that way when you can actually do this more integrated approach.

And nowadays people would look at that. And I even had PhD students after reading the paper, read that and be like, how is this groundbreaking? Everybody just does this. And I'm thinking, that's the point. That's the point. That paper was basically saying, everybody should be doing this. And in 2026, it's very easy to say, oh, but everyone's doing this, whatever. So just the idea that those communities should be working together and integrating what they're doing on a foundational level, I think was a tectonic shift, intellectually speaking, for the Viz community and the database community. And I think that's also why Tableau as a company had this like wild commercial success soon after that, because those communities just weren't connecting in a way that that company was able to kind of put them together at the time. So to be fair, there were definitely companies before and after who've done great things that are kind of in a similar spirit, right?

I didn't realize though, there was a research paper associated with Tableau and a lot of it. That's pretty crazy to hear.

I do find it so fascinating, how we kind of build these boundaries around, this is the area in which I work, or this is the area in which we work. And then it becomes almost heretical to step outside of it or combine it with something else. It's very difficult. Each community builds up its own kind of standards, its own vocabulary. And that's how you generate these shibboleths. This is how I advertise myself to other people, so they know I'm a serious database researcher or a serious Viz researcher. And then when someone comes out from the outside, and they don't know how to like signal all of these, like, oh, I'm a serious whatever. And just so can be like so hard to like, yeah, like that. I guess I never really understood why like interdisciplinary or cross disciplinary work was so hard, but it's not really about the work. It's about the people and the way they define themselves and like the communities.

Absolutely. Yeah, I think so. And lots of times a community defines itself by the problems it chooses to focus on, that community chooses to focus on. And that work and then also just my, I like to think the work that I focus on in my career is trying to break down some of those walls to reclaim some of what we lose by those, you know, those boundaries that we those artificial boundaries that we put up to kind of just define and protect our communities in some ways.

Do you feel like today you mostly are the are the conferences you go to today mostly split still like, like bucketed in one of the two areas? Are there many sort of like, cross disciplinary conferences for sort of the cross-disciplinary conferences for sort of the HCI viz and computation stuff?

That's a really great question. So as far as conferences I publish in, um, I will publish wherever I believe the work is a good fit, that could be an HCI conference, that could be a visualization conference, that could be an AI ethics conference, that could be a, you know, more programming languages oriented conference or a database conference, I think to me, the most important thing is that community cares about the work that we did, so this would be different if I was working really hard to get tenure because I would want one community to know me well, so I'd probably want my home base to be in one or maybe two communities so that when, you know, letters go out recommending my tenure case, they're

saying, I know Leilani, she does great work in X.

If you're too diffuse in your portfolio, um, then people can't write those kinds of letters because they'll be like, who's that, Leilani, who's that? Or Leilani, I don't really know her work.

So there's that danger, but if you can avoid that danger, I think it's better to focus more on where the work will be best received and have the most impact rather than worry about, you know, oh, I published all my papers at this.

That being said, tragically speaking, I can't jet set around the world all year. I have to be practical about what conferences I actually go to. I try to make sure when possible, try to make sure I go to at least one like Viz or HCI conference and one database conference a year. The past couple of years have been weird and I've been prioritizing sending my students to conferences instead, but, um, yeah, typically when I attend a conference, I'll try to go to one of each.

Code switching between communities

And, and are you sort of conscious of like code switching when you're writing like a database paper versus like a HCI paper? Yes, I am deliberate. Uh, and I do code switch between communities because the things that they talk about, the terminology they use, the things they emphasize are different.

That being said, with AI, things are blending a little more. We're sort of ending up in similar places more so than I've seen in the past because we have kind of similar goals we want to achieve, so I think things are blending more now, but I do have to code switch when I talk to different communities.

And, and when you say AI because of AI, you mean like the problems it faces, not that people are using AI to write all their papers now and hence all other papers seem the same regardless of discipline?

Yeah, I mean more that, uh, say you want to use AI to help people achieve data science tasks, like complete data science tasks, analyze their data, things like that, database researchers looking at that problem and visualization and HCI researchers looking at their problem will end up in similar spaces because there's things like how exactly do we prompt the models or design infrastructure around the models to facilitate this, when a model manipulates the data, how do we communicate that to users in a way that they can feel comfortable that they understand what's going on and they can trust it. And then also if we want to empower users specifically to analyze the data, how can we help them do that in tandem with the model or maybe co-plan and coordinate with the model such that, you know, they achieve the goal that they want to achieve in, with hopefully, you know, minimal effort.

So with those things in mind, right, database people are starting to have to worry about interfaces. A bit more and transparency in the behavior of the system because users are a little less, uh, trusting of black boxes and want to know what's going on. And also when a mistake happens, it's not super obvious why the mistake happens. So users are going to need more transparency as well on, oh, why did this AI powered or LLM powered operation do this thing I didn't understand. And then also on the flip side with these people, right, uh, being able to provide enough context to the model can very easily and quickly become a data management problem as well. So there, it's interesting how those two sides are blending, uh, with the shared interest of maximum utilization of OLS.

Disruption in data visualization research

I mean, one thing I'm curious about is, I mean, is there, is there a sense kind of in your, in your community that, uh, your research community that, that, you know, your field is going to experience like the same type of, you know, or different, you know, certain type of disruption compared with more general computer science or, or software engineering. Like I think, you know, software engineering is clearly like in a state of upheaval and disruption, but I imagine like in, in data visualization and interactive data that, that there is disruption, but perhaps we don't know like what things are going to look like on the other side, I know, at least in the business world, there's a lot of discussion about like headless BI and and, you know, semantic layers becoming more important, like trying to give agents like a simpler language to talk about, you know, relationships and data models and describing data visualizations and, and, and things like that.

And like, I feel like databases have become even more, you know, even more important because they can't be, they can't be vibe coded quite so easily. Whereas I feel like we're going to see a lot more like just in time, you know, dashboards and like just in time data interfaces where you aren't necessarily looking at, you know, something that was developed over a long period of time. It may be something that was stood up on top of, you know, stood up on top of duck DB or on top of one of the interactive, you know, data visualization frameworks, you know, and, and like a, as like a, you know, throwaway prototype. And, you know, in the past you, you can't, you couldn't imagine, you know, building some interactive data application only to throw it away and then rebuild it, you know, the next day. But now that's something that, you know, you could build these like just in time interactive data interfaces. And so I feel like that is going to lead to like new interesting avenues of research, like how to incorporate, like how to incorporate LLMs into the, like, you know, essentially into the process. So.

Absolutely. I think we're already experiencing a massive shift in the way people do data science work with the introduction of large language models.

One easy example I can point to is in chart generation and visualization recommendation, a lot of, a lot of my prior work has been in visualization recommendation and automated methods to support that. And now we always have to answer this question of, oh, well, how would LLM solve this problem or how would someone use an LLM to, you know, generate a chart or generate code to generate charts or even generate dashboards? So when we try to introduce new ideas in that space, we always have to be thinking about it.

And also practically speaking, um, a lot of people I know, researchers, folks in industry, they'll just very quickly go to a model and ask it to make charts for them. So just on a practical level, people are using LLMs in a way where they would traditionally use maybe visuals and stuff like that, and also even tools like, uh, data science tools themselves are doing it as well, right? Just automatically using, providing AI recommendations. So I think huge, huge change.

Statistical consulting in the age of LLMs

I can spotlight one ongoing project in our lab around this because we are curious, you know, what this shift looks like. So Will Wong, a PhD student in our lab, who's co-advised by myself and Jeff here, he's investigating this problem by interviewing, um, statistical and, and data analysis consultants at universities.

Because traditionally pre-LLM days, if you're a scientist and you weren't totally sure how to, you know, analyze and statistically test your data that you collected for an experiment or something, your university, there's a very good chance, like you could talk to the statisticians there, or there's even a consulting office you could go to and then a consultant could work with you to figure out what questions you're trying to answer from your data, whether you collected the right data, what statistical models and tests make the most sense, how to process your data properly for it and so on, but you can imagine with LLMs, like statistical consulting must be seeing a humongous tectonic shift, like experiencing a huge shift in their like client base and how they're utilized, how consultants are utilized. So, uh, we are wrapping up an interview study with consultants to see how, like their perception of large language models and how it's affected their jobs. And it's been, it's been really interesting.

Yeah. It's so interesting to hear that. Yeah. That, that avenue opening of like, you're mentioning like just being able to ask an LLM to generate a visualization. And in the context of your really thoughtful, careful work on like visualization recommendation, still thinking critically about what's the quality of that recommendation, um, and like who's asking for it. And it sounds like you're also saying like, what happens to the people who were, who they were going to before for that advice, all those kinds of elements.

From different angles, we've been tackling this problem. So Will, he, the PhD student I mentioned, he also had a previous paper on evaluating the kinds of recommendations at the time that models were giving when you asked them to generate charts or to complete a partial visualization design. So if you had a partial design, you wanted the model to finish it, what would it recommend, or if you wanted the model to just recommend charts, you know, what, what do you get from it?

Um, at the time, so to be fair, this is a paper from 2024, it's 2026 now, models move fast, update quickly. At the time, his work found that what the models were recommending was mostly fine, but there are some cases where they would recommend weird stuff. Like they would recommend visual encodings that didn't really make sense for the context.

And I think kind of the big takeaway for me was that sometimes the models were recommending designs that didn't seem to align with the data that we have in our community. So as science, as visualization researchers, as scientists, we've collected data. We've done tons of experiments seeing what kinds of business design decisions make sense, which ones don't make sense. And it just didn't seem like models, like what they were learning on was not that, it didn't seem like it was that, at least not most of the time. And for proprietary models, who knows what exactly they're learning their business recommendations from, but it seemed like the answer was not us sometimes anyway.

And for proprietary models, who knows what exactly they're learning their business recommendations from, but it seemed like the answer was not us sometimes anyway.

The shift toward integrated Viz tools

Do you think this is changing how the Viz community, like the types of tools it makes, like my recollection, which is now very out of date is that, you know, people would tend to produce kind of like, you know, polished UI point and click kind of interfaces, but obviously like an LLM is going to have a hard time driving that. Do you, so do you think this is going to push the Viz community to produce tools that are like plain text or JSON or code-based that then the LLM can generate the visual specification?

It's a good question. So kind of before, during, and now like this before and during the AI wave, I think we were already in some ways transitioning away from these kind of standalone Viz tools, at least in the specific areas I work in. We're sort of transitioning away from these standalone Viz tools and more towards like modules and things that can integrate into stuff like computational notebooks. And I personally feel like that's the future, having a standalone, unless you're providing an environment that other tools would integrate into. I don't think it's really to your advantage to have a standalone tool that you create as a Viz researcher, because then you have to argue to everybody why they should leave whatever environment they already use and use your tool instead and then export whatever your tool makes back into their usual workflow.

So, yeah, I mean, that's fascinating because I mean, I haven't, and not that I was ever particularly active in the Viz community, but when I was active, that was like 15 years ago. And that was the thing that struck me, like, why would someone use your cool new visualization when they've got to do it in a completely different computational environment? Like, and so it just felt like the Viz community didn't actually care about people using their visualizations, which is, you know, like fine, not all research has to be like end user focused, but it did feel like weird to me. So that's, that's neat that people are more focused and like, let's integrate this into where people are already working.

You still see a lot of standalone tools, just practically speaking, cause it's easy for a grad student to make a research prototype that way than to build a, like Jupyter plugin or something, that being said with large language models, Jupyter plugins are way easier to make now than they used to be. So I think there's less of an excuse now and that integration, cause I don't, I think there's more of a question around computational notebooks years ago, like earlier in my career. Nowadays, it's like a no brainer computational notebook environment. Like why wouldn't we use it? So I think the Viz community has been forced to adopt it more.

Yeah. It's funny. Cause yeah, cause computational notebooks are like such a, like in some ways and like, I can, you know, that's such a terrible environment, but like that, like clearly like the least worst environment, like everyone knows them by just this great, like, you know, ground floor that everyone can rest on. But I'm sure I like, I can just imagine like the Viz community looking at them and being like this, these are awful. Like there's so many problems with them that.

Kind of like malproduce and the database community.

If you can't meet them, join them or come up with a better version.

That reminds me of this advice I heard years ago, like how, like if you want to, if you want to pick some, um, like machine learning algorithm to use in your actual applied work, like how do you, how do you do what, how do you figure out which algorithm should you should use? And the advice was like, go read like three of the top recent papers and then look at what they benchmark their method against and they use that. Cause like that's the gold standard. It's so true, actually. That's a smart way of picking a good commercial tool to use.

Responsible data science and favorite Viz studies

I do, I think it'd be nice too, to talk about the responsible data science stuff, which I know is a lot more of your recent work, but I figured maybe before we get to that, I'm curious if you have like a favorite Viz study you've done. Like, do you have a study of yours? That's like a personal favorite that you've done on Viz?

I would say there've been like takeaways from different studies that I've done that I've really enjoyed. I don't know if I would say, Oh, that whole stu- everything about that study. I just love it about everything else and all the rest.

You have some favorite takeaways though. Yeah, I have some favorite takeaways.

So one funny takeaway from a study we did in 2022, I believe. That was the 2022 CHI paper. So, uh, this work was in collaboration with Michael Carell.

He was at Tableau at the time, but now I believe he's at Northeastern. But the, it was around visualization recommendation. And our idea was to see, okay, how would people like automated recommendations versus like human curated recommendations and what arguments are people making in terms of why they like one versus the other.

So what we did was we picked up some, picked out some commercial tools like Tableau, Power BI and such, had them generate recommendations for visualizations of public health data. And this data was specifically a subset of data from NHANES. So the, um, it's basically a national health survey. I'm not going to make myself regurgitate that acronym right now off the top of my head, but it's basically a national health survey data from the, from the, uh, I think from the CDC or, or the National Institutes of Health, one or the other, maybe the National Institutes of Health, but we had this health survey data. We showed it to, um, students, professionals, and researchers in public health.

And we showed them recommendations from, you know, automated tools and also from just people crafting recommendations and the professional, like not the, the participants of our study generally tended to prefer the human created recommendations because they felt that the human created recommendations told more of a story. There's more of a story behind why that recommendation was generated.

And at the time, the automated tools, they just, they generated pretty overly simplistic recommendations. You could tell they were just putting variables together and then generating a chart. And so that like is a, a fun, like favorite takeaway of mine. Of course, this is like pre-LLM. So maybe the automated tools would do better now. So we should redo that experiment.

Yeah. It's interesting to hear that it's people, they prefer, yeah, recommendations from people. It sounds like maybe, was that surprising to you?

What I would say was surprising wasn't even necessarily at the running the study stage, but at the experiment design stage, when we were looking at the recommendations, cause we could also see it before we ran the study. We were like, how are these tools generating these recommendations? This seems really generic. Why would anyone want this? Maybe we should have given that feedback to the tool creators, but yeah, that to me, it was always surprising.

But maybe it's because I'm a, I'm a researcher who focuses on visualization recommendations, so I could see it when I was trying to use these tools. Oh, you know, these recommendations are not like, I don't understand who they're helping, but it doesn't seem that they're helping anybody. It sounds like the thing that really struck you was the, as you were running it, you were really struck by the kind of difference between the automated and the humans.

Yeah, I was just struck by like how some overly simplistic the automated recommendation methods were at the time. Like, I guess they wanted to, from a commercial standpoint, I can understand wanting to generate recommendations that will be, you know, consistent. They'll always work. They aren't going to make any customers unhappy, that kind of thing. So I can understand strategically why you would do that to, but as a researcher who, you know, likes to do, push the boundaries of what computationally we can do, I was thinking, there's no way that someone would pick his recommendations over the human curated ones, and I, I hope it was not a self-fulfilling prophecy. I like to think I designed good experiments, but we, we pretty much saw that in the study results.

Yeah. It is interesting to hear. It's like, you kind of see it coming, but you have to run, run it through. Yeah. Yeah, basically.

The latency manipulation experiment

And then one more study I'll highlight that was really fun. Actually, it was from the tail end of my dissertation work. Basically you wanted to see, okay, well with recommendations, there could be this issue of performance where let's say you're analyzing a large dataset. What if the thing people look at is the thing you can recommend the fastest? Basically the thing they see first. How could that potentially introduce manipulation of people? Like what, what conclusions they might draw or what findings they might find in their data? So what we wanted to do was design an abstract experiment to try to test this. Could you, in a sense, manipulate what people found in a visualization purely through latency?

Like how quickly or how slowly people might see the data or different parts of the data. So I designed this like really abstracted version of this experiment. I created this image collage and asked people to explore the collage of images. And then they were looking for certain, a certain target image. I actually put two targets, like two copies of that target in the collage. But the goal of the task was to only find one. And you can imagine kind of like Google Maps. Hey, you sometimes say you're on your phone and your data speed isn't that great and the map takes a while to load. Maybe it loads in kind of patchy. Imagine something kind of like that in this experiment, but it's an image collage so some images appear faster, slower than others.

So I designed the latency for the images such that it would look kind of random. But in fact, it was definitely not random. I was manipulating how quickly certain images would appear. And I was trying to make it such that I only use latency to try to incentivize people to find one target over another in the collage.

And the, the experiments showed that if the latency was pretty low overall, like relatively low, it didn't affect people as much. And then, but gradually as you ratcheted up the latency, like if you made it the tile, the images load in slow enough, you could definitely manipulate people to find a certain target in the collage. So it was just showing how latency matters, how you design a system matters, but also how you recommend things matters. If you do it purely based on performance and don't pay attention to what, you know, what you're prioritizing, like if you're, if you have options to show A or B and you always make A show up faster, depending on how much faster A shows up, you might be incentivizing the user to just always look at A and never look at B.

If you do it purely based on performance and don't pay attention to what, you know, what you're prioritizing, like if you're, if you have options to show A or B and you always make A show up faster, depending on how much faster A shows up, you might be incentivizing the user to just always look at A and never look at B.

That seems like super relevant in the era of LLMs again, where like latency is becoming so important again. Absolutely. So that to me was like a fun experiment that I enjoyed. I really enjoyed the design of the experiment. The takeaway was pretty straightforward in my opinion. I guess one thing that was surprising about it was people kind of thought of latency as a sort of binary thing, like, Oh, there's either too much or too little. But that experiment was showing it's actually kind of a gradual effect. And the more latency you have, the worse the effect becomes for people that get, become kind of biased. And the, I guess the latency between two things being relative as well, I guess, where two things are competing.

Yeah. Like in an image collage, there are all these different images you could be looking at, but your eye and your mind is sort of drawn to the stuff that shows up first and if it shows up way earlier than everything else, you're just going to kind of predicate your exploration on that.

I feel like it's so helpful to hear the like surprising pieces of people's research, like what stood out and kind of whether it was the, I think it's interesting to think about too, like the thing that could stand out to someone or that you might like might be the designing of it or the like what you saw early on. And, and the takeaways. So yeah, I really appreciate you sharing. I know, I know we're getting long, but I definitely want to get to the responsible data science because I know it's what you're working on now. Could you, could you tell us a little bit about that work and how you got into it?

Responsible data science and behavior change

Yeah. So responsible data science is this giant umbrella. Lots of people work in this space. At a high level response, the whole point of data responsible data science is the idea that we want the outcomes of data science work to be responsible. We want that people are not discriminated against. We want that people are not harmed by the outputs of say models, large language models and otherwise. We want people to be treated fairly.

So everybody who wants that and is working towards that, they fall under this responsibility science umbrella. What I think is really cool about the work that I'm doing, and this is in collaboration with a few professors who I want to shout out, so I want to shout out Emily Wall, kind of my longest standing collaborator on this work. Professor Amy Zhang, who Amy and I co-advised an awesome PhD student working in this space, Tiana Barrett. And then also Professor Yuichi Shoda. So Yuichi is a faculty member in psychology at UDub and without Yuichi, we wouldn't be, I wouldn't be where I am now with these projects. So I think his perspective has been really helpful too.

But the thing that always struck me about responsible data science work, where I think it's changing a bit more now, but years ago when Emily and I started talking about this problem, what struck us about it and what struck me about it especially is how it was always sort of treated as a algorithmic problem, a model training problem, or a data cleaning problem. The problem was always someone or something else. The problem was never us.

The problem was always someone or something else. The problem was never us.

And I was just thinking, wait a minute. So who does data science work? Is the data science worker the model? Is the data science worker the data or is it me and you? So if we're going to, you know, try to make data science work more responsible, should we be emphasizing the artifacts or should we be emphasizing the people doing the work?

So what came out of that is this idea of behavior change interventions for responsible data science. So the idea that we should not just focus on whether or not a data set is responsible, a model's responsible. We should also be focusing on, you know, are the practices of the data scientists responsible and how does that contribute to ethical or unethical outcomes in data science? So that's the perspective that we brought to the community.

Because initially people weren't really thinking about the problem in this way. They weren't talking about it in this way. People are actually kind of skeptical too about it. It took us a bit to actually publish our very first paper in that space where it was kind of a mixture of a literature review and vision paper describing what behavior change would mean in the context of responsible data science and what kinds of tools in psychology and in computer science that we could use to actually achieve that goal of trying to help people shift their behaviors.

And then since then, we've had a couple, a few more papers on that, uh, line in that line of work where the most recent one is actually a paper at CHI, so top HCI conference, where we actually evaluated two existing behavior change intervention strategies in the context of a controlled, uh, data modeling task. So we asked people to train a machine learning model with some provided data that we gave them during the experiment.

And then we, they experienced, uh, I think they experienced two different intervention strategies. So one was we primed them. So we had them read a little prompt that hopefully motivated them to be more responsible in their actions and training model. And then a separate intervention where we actually just loaded, um, like a mo- a modular plugin into the notebook they were using to train the model. And they could use that tool, that specific tool to check the fairness of their model and improve it.

Yeah. Yeah. So sorry, I have it here. I was trying to grab the, but this is, um, evaluating behavior change interventions for responsible data science. Is that? Yes. Yeah. Yeah. Nice. Evaluating behavior change interventions for responsible data science.

Yeah. So a cool outcome of that work was that priming people to think responsibly did shift their behaviors. The, they did have an increase in the range of responsible activities they did during the session. But the fairness of their models did not actually get better from a significance perspective. We did not notice a significant improvement in the fairness of the models. So just motivating people to do better, basically just telling them to do better, and them feeling like they want to do better doesn't actually lead to anything getting better. So things change, but not necessarily in a measurably better way. So giving them tools to actually show them like this is what better looks like in terms of outcomes and pairing that with the motivation, that actually works.

I think that behavior change framing is like so interesting and like powerful. Like it's not something, I don't know, like I think in you know like computer science and statistics it's so easy to get locked in on like the algorithms and the mathematics and forget like oh actually it's humans doing all this and we have these levers to pull on how the humans use our tools and that's just as valuable. Just as powerful.

Yeah. Yeah. What was the tool called? Is it Equitas? Oh, Equitas. Equitas. Yeah, okay. Yeah, what does that look like? It runs inside the notebook, is that? It does run inside the notebook. It shows you some bias metrics and you can pick specific subgroups in your dataset that you want to compare against. For example, a dataset that we asked participants to train models with was sort of a classic like loan application datasets. So a demographic group you could compare would be like men versus women and their loan applications. So you could see if your model would be say biased against women and be more likely to reject their loan applications.

So it's like versus the prime, which is more of a nudge. It gives them some like tools to kind of implement some like checks or... It's more about framing. So the prime is really just a prompt that encourages you to think like you could think about saying, hey, with the loan, the credit loan dataset, it's possible that women are more likely discriminated against than men and more likely to have their loans declined. So you could say something like that and like, oh, please keep this in mind when you're training your models.

And as I'm understanding, it's like people will... The prime actually does nudge people in that they kind of, they've both like self-report. They're like, oh, I did follow these kinds of like principles more.

Yeah. People would say, oh yeah, I was definitely more motivated, like being reminded of that or, oh, as a woman, I can resonate, that resonates with me and I would not want people to be discriminated against in that way. It definitely motivated people. And we did notice a behavioral shift. But I think the important thing is just inducing a change in behavior does not guarantee the outcomes change. You have to induce the behavior in a deliberate direction.

inducing a change in behavior does not guarantee the outcomes change. You have to induce the behavior in a deliberate direction.

Yeah, gotcha. Like, I guess, like provide the skill or capability to kind of actually produce a fair model. Yeah. Giving them the capabilities to measure the fairness of their model. And then also to fix the model when it's not at the level that they want or if they want it to be more fair.

Yeah. Gotcha. Yeah. It's super interesting. And as I understand it too, the, like, the evaluation of, like, there's two parts, as I understand it, like the responsible data science, you have the framework for it. But this paper is really interesting that it's also attempting to evaluate it and provide like a behavioral experiment. And I think, as I recall, I think the paper says it's like one of the first controlled experiments in kind of like evaluating.

Yeah. I'd be curious to hear a little bit more about that.

Yeah. So behavior change, behavior change for responsible data science is just, in my opinion, understudied. There should be way more people than the, me and the faculty I mentioned studying this problem. And there are some people studying this problem. So I'll acknowledge, like, this is not an empty bucket, but there should be way more people studying it. And I think part of the challenge in studying this problem is that we are not necessarily well-equipped with the tools that we need to tackle these kinds of behavioral problems, but there are other communities that have studied it for a long time, like psychology.

Behavior change is a really important aspect of psychology research. And so what we are trying to introduce through these papers are strategies and tools that we could adopt where we maybe, we could even collaborate with researchers in psychology, like UHE's collaboration with us is an example of that, where we don't have to come up with all the answers as computer scientists. And in fact, we shouldn't, because a lot of us are not behavioral scientists. A lot of us are not psychology researchers. We should be collaborating with other people and we should be learning from them to develop solutions. So all of this work is really trying to build infrastructure intellectually and, you know, mechanically speaking, like with designing experiments to measure success, to see, you know, what could behavior change interventions look like in this context if we were to build more of them and, you know, what are the right things to do and what are the kind of the pitfalls we should avoid in those designs. I saw a lot of interesting measures like cognitive load and things like that. So it's cool to see kind of the full array of like instrumentation, you know, in this experiment.

Yeah. I'm glad you mentioned cognitive load too, because an important thing to acknowledge in that work is that we're asking people to literally do more work than they would normally do when you're asking them to do data science work responsibly, which also means introducing a cognitive burden on top of what they already have to deal with as a user of data science tools. And the load measures from that study show that, reflect that. It's not shocking, right? It makes sense that if you're asking people to not just worry about model performance, but also whether it's discriminating against people, that's more work that they had to do. But it was cool that those measures in the paper show that. Yeah, it's a really cool approach.

African data ethics framework

And then I know the other one I saw that you put out recently was, wait, what's this? The African Data Ethics, a Discursive Framework for Black Decolonial AI. Yeah. I'm really curious to hear more about that paper as well.

Yeah. So I definitely want to acknowledge the intellectual leader of that work was Tiana Barrett. So she's a PhD student here at UW, co-advised by me and Amy Zhang. That was kind of her first major project as a PhD student here at UW. And I felt like she did a fantastic job with it. And so what's interesting to me about that work is this idea that, so, you know, in responsible data science, we care a lot about what it means to do something right and what it means to do something wrong, ethically speaking. And a lot of how various communities try to write that down are through frameworks, ethics frameworks. What we noticed is that in these ethics frameworks, you have them for a lot of different communities, but it seemed like African countries were being left out of that conversation of what, like, what is right and what is wrong in the context of interacting with, engaging with African peoples. And it seemed like we could develop a framework to reflect the values and the ethics of different African countries to fill that gap. So that was the idea behind that work. And Tiana worked really hard to take into account African, like Sub-Saharan African philosophy, ethics work, kind of thought leaders in terms of how AI should be incorporated into their cultures, things like that, and kind of putting it all together to review, okay, if we were to create one framework to exemplify these principles that should be upheld, what would that look like? And what gaps are we filling compared to other ethics frameworks? Like what, which of these principles might be covered in current frameworks and which are not? And I would say anyone who's really excited about it, please reach out to Tiana. She loves talking about her work and that paper. And a lot of people have been really excited about it. So I just encourage folks to reach out.

Do you know what y'all might do next as next steps building on the paper?

Yeah, that's a great question. So one thing that really bothers me about ethics frameworks is that they're so hard to use in a practical sense. I could go out, read a bunch of books, learn all there is to know about these frameworks, but then how do I actually apply it to my data science work? And then also a more recent project that Tiana has been working on is this idea of, well, how exactly do data scientists operationalize their ethics and their values in their own work? And the answer is it's really dependent on the environment that they're in and the opportunities that they're afforded to express those values through their work. And some people haven't thought as deeply about it as others, but even people who think deeply about it, depending on where you work, you might need like outside projects to be able to do that, to really closely align your values with the practices in data science that you follow. Because your company might be hostile towards it. They might not care. Like you better meet your key performance metrics and I'm not going to give you any extra resources for this other stuff where it might be that resources get cut back at work. So then the time and the energy and resources you had to do some more of the responsible aspects might be reduced. So it's very situational and there are a lot of factors involved. So what we're interested in is, well, for the question of if it's hard to figure out what your ethics are, what values you want to uphold through your data science work, could frameworks help you and could we make frameworks easier to use in that direction? Another question is if there are these barriers, these environmental barriers to helping you, you know, practice data science more responsibly in the way that you want, from a technology perspective, what can we and what can we not do to support that? You're saying outside projects, they might need outside projects because they just don't have the resources. Yeah. Or the bandwidth or the company's just not interested or it doesn't want them to do that work within the context of the company. So then maybe they may have a passion project outside where they're sort of, I don't want to say atoning, but like allowing themselves to, to focus on where their heart is.

Yeah. It's interesting to think about too, the, in terms of responsibility that I guess, if you're a data scientist within a company, the question of like, what is my responsibility? And if I don't have resources in the company to kind of like meet that responsibility, should I maybe do, do I need to put in work to kind of like build the skills to, to behave responsibly? Or even just how hard should I fight to try to make these resources available so that people can perform their work responsibly in the way that aligns with my values? Cause the organization has values and then the individual has values and there's this question of how they intersect. Yeah. That's a great point.

Rationality and responsible data science

Heather, have you run into much discussion around or thought about responsible data science?

Yeah. I think we, we think a lot about, like I don't, like we don't use like responsible, but we think about a lot of these similar things in particular, like, you know, for me, I think a lot about the, like the pit of success and like, how do you kind of guide people into doing like the right thing or the better, like making the right thing or the better thing, like easier than the hard thing, easier than the bad thing or the worst thing. And I think it's fascinating to like, we now have a lot more levels to pull in terms of like prompting LLMs and giving them tools and having, you know, different agents review your work afterwards with different, with different lenses. So I think it's like, it's an exciting time to be thinking about how can we make these frameworks more actionable for, for, you know, real data scientists who of course want to do their best, but have so many other pressures on their time.

Yeah, absolutely. I was going to say one direction I'm really interested in moving in the future is this idea of, yeah, it might not necessarily be right versus wrong in this kind of more nuanced space. Something I think we're running into and that we see is that inaction might be the kind of the biggest problem, inaction along different dimensions, because it's pretty clear, right? That if data science, data scientists and engineers and so on do what they believe is rational or what they believe is like reasonable and are not actively doing actions where they're like, you know, I should be doing additional things to make this as responsible as possible versus, oh, I don't think I'm doing harm. So I'm, I think I'm fine. We still end up in all sorts of bad situations. So it's not that people aren't rational or people aren't reasonable. It's that the rational and reasonable, like rational behavior is not in the, in the current context, the definition of rationality might not be ethical. So if you define rational as responsible, you end up in a different frame, which might lead a totally separate line of decision-making that we come very different from what we do right now. So something I'm, Emily Wall and I are interested in following is this idea of rationality in data science and where it might get us into trouble and how other frames, like other definitions of rationality might help us kind of break out of that cycle. Wanting to make money is rational, for example, but can lead us all sorts of directions. Sticky situations.

the definition of rationality might not be ethical. So if you define rational as responsible, you end up in a different frame, which might lead a totally separate line of decision-making that we come very different from what we do right now.

Yeah. And it's not necessarily bad, right? But sometimes the outcomes are bad. So the question is like, how does that frame, that definition of rational get us in trouble?

Do you have advice for people maybe starting out in data science today or recommendations for them? You mean for like responsible data science? Yeah. For responsible data science.

What I would, what I would say is you don't have to like read a bajillion books on it or anything, maybe pick one book, like Data Feminism and check it out. And you don't have to necessarily commit yourself to doing everything in that book, but just kind of, I think it's useful to know what's out there and then you yourself can decide how you want to incorporate it. And then I would say for each tool or feature that you're going to use to automate your work, see if there's another tool or feature that you can use to evaluate it from a responsible perspective. Like if you're going to train models, see if there's already tools out there that could help you check how fair they are. If you're data cleaning and you're going to do some automation, see if there's anything that can help check for like bias.

It seems like such a helpful rule of thumb, like for each thing you use, find a thing to evaluate it as like such a practical piece of advice to kind of like try to counterbalance. Like an easy

thing to kind of like nudge a person. I feel like that, that strikes me as really helpful for being able to kind of like put this into an operation. Yeah. If I was starting at the very beginning, that's what I would do. I would think about, you know, almost like unit testing. If I already do the equivalent of that for any tool I want to add to my toolbox, you know, what checks and balances can I just automatically put in place as I add to the toolbox. Hadley and Wes, any other questions?

Just this, I just, I mean, I just really appreciate your like thinking about, you know, responsible data science. How do we kind of like steer people towards it? You know, not about being perfect, but how do we help everyone just get a little bit better? I think that's such a, the framing around by behavior interventions to just feel like it.

Absolutely. Agreed.

If you're a fan of like the good place, I think of it kind of like that. The good place? Yeah. There are no perfect choices in life. You can only, you know, do better. And that's just such an antidote to like paralysis. Like, you know, you can't, you're never going to be perfect. You're never going to do the best thing you possibly could, but you can do a little bit better than what you're doing right now. And everyone can do that.

Exactly. It's so, it's relative. I think of responsibility in relative terms. There aren't good people and bad people.

I think of responsibility in relative terms. There aren't good people and bad people.

Well, Leilani, thank you so much for coming on. Sorry. I've been so frank. I mean, the thing is, I think you have so, so many interesting studies that span like computation, cognition, and ethics. Like I think that I got too excited in the the sauce, but honestly, it's been so nice having you on and hearing from your work at the intersection of so many interesting areas and really focusing on, or being sure to bring in also the human component into so much of it. And to think about the ethical side of it has been so inspiring to hear about. So yeah, really appreciate you coming on the test set. And thanks so much for talking.

Yeah. Thank you for having me. It's been a blast. I've loved talking about all the topics we covered today, especially responsible data science. I actually don't get that many opportunities to talk about it. It's not that popular of a topic in industry right now. Well, it's super exciting. I think if people, it's cool that people both have the framework and can read some of the experiments you've done, you know, evaluating it too. Yeah. Awesome. Yeah. Thanks so much. Yeah. Thanks for being here.