Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time.
If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
We can't wait to see you there. Will, thank you so much for being here. I am so excited to talk with you today. Will is a Senior Advisor of Innovative Statistics at Eli Lilly, and I would love to know what that job title means because it's one of the coolest job titles I've ever heard. Can you tell us a little bit about what you do and something you like to do for fun?
Sure, and thanks for having me. It's really amazing to be here. I work in a group of several hundred statisticians at Lilly, and our group is a central tool development, method development consulting role. Statisticians are organized by therapeutic area, like cardiometabolic health or neuroscience or autoimmune diseases, et cetera. We're kind of like an everything central group, and we spend more time. We're not necessarily saying that we're better at methods and tools, but that's where we spend most of our time. Whereas one of our colleagues would say, I am developing this simulation to design this autoimmune trial, in our group we say, okay, what are the different kinds of simulations and models that people across the company are developing? Let's standardize and make these implementations rock solid so we can roll them out to everyone so other statisticians have an easier time doing their work.
Senior Advisor is really a fancy way of saying statistician, and those are just in general ranks across the company. We used to have job titles that said senior scientist or things like that, very general, but all it means is statistician in my case. As far as something that I like to do for fun, over the last five years or so, I've been getting oddly into backyard ecology. Just from walking around my local park, and eventually I decided to join a group that walked with the head naturalist there. I started learning about different kinds of common native and invasive plants, and just learning, oh, that over there, that's bush honeysuckle. That grows in monocultures that take over the park and are doing ecological damage to these pollinator pathways and watersheds that we have around here. That lit a fire under me. Now I volunteer with a bunch of people locally, and we have this little meadow that we're working on removing invasive species from the park. We're seeing all kinds of really nice native plants coming up again. We're gradually seeing diversity return to the area, and we're creating this little oasis wherever we can. That's been really fun to get my hands dirty with that, and gradually see the impact build over time.
I love that. This is one of my passions, too. For anybody who knows me in real life, I'm a hashtag meadows, not backyards people. My backyard is a native paradise for all kinds of Texas weeds. It's amazing. We have a lot of invasive species, too, that we're trying to get rid of and make sure we have spaces for all of our wildlife to live. Oh, we're going to have to talk later, Will, because that's really exciting to me. Okay. I see some other people in the chat, too, who are like, yes, this is exciting, or who got their start in soil ecology. Also, Michael said that there's registration for the Organization of Fish and Wildlife Information Managers annual conference, which is open. That sounds very, very interesting, too. I love attending conferences that are about information management, data management, things that are outside of my purview. I always learn so much, especially our pharma, our medicine. Those are places where you don't have to be in pharma or medicine to learn a ton. I highly recommend them to everybody.
Okay. Well, you did not mention, I think, the one thing that I really want everybody to know. Can you talk about your relationship with the targets package? Speaking of making things easier for a lot of people to do.
Origins of the targets package
Yeah. That's a great story. I'm glad you brought it up. Because I was, in 2015, like many PhD students, I was struggling to get my dissertation done. Of course, a lot of that is the writing, but for me, it was, and a lot of others probably on this call, it was a computational struggle as well. Because I was trying to fit this complicated Bayesian model on a collection of simulated and real agronomy datasets in genomics. These are datasets that, in terms of big data back then, it was just kind of medium data size around big fitting datasets that were kind of like 30,000 genes by 16 observations or something. And the computation on that was quite intense, though. It was a long, hours-long Markov chain Monte Carlo algorithm that was fitting each time. And I would struggle going back and forth between sending these off for computation to our local servers, and then midway through the computation realizing, well, something's wrong. There's a bug here. There's something I want to reorganize here to support the writing. And it was just this constant back and forth of trying to keep results up to date and waiting a long time for them to finish, and then getting disorganized and confused really quickly.
And so I got really interested in pipeline tools. And credit to my advisor, Jared Neamey at Iowa State, because he said that, well, a lot of the struggle is needless. A lot of the struggle is needless. You should be using a tool like Make, which of course is a tool from the 70s to manage compilation of C and Fortran programs. But I think then people like my advisor, who were computationally really savvy, were just starting to realize that Make and its main operating model had uses for reproducible data science to both only run results when the rerun is actually necessary, meaning that if a result was up to date, if an analysis is up to date with this upstream code and data, why not just skip that and only rerun the models whose data encoded changed?
And that would have saved me a lot of time then. And there was also the mental model of a workflow, a mental model of a data analysis pipeline. And I think that more than the computation, more people in the statistics community get value out of having a mental model, a graphical mental model for what exactly is going on in a research project or simulation study or anything of that nature. And so at the time, we had things like Make, but we didn't really have something that was idiomatic to R that fit the particular concerns and sensibilities of our users, a reliance on functions instead of scripts, integration with literate programming, and those things that were really important that just weren't in language agnostic pipeline tools.
And so at the time, Rich Fitzjohn, who is also really active in our community in ROpenSci, he's at Imperial College London, I believe. And so he was working on a tool called Remake, which was, in my mind, a first stab of this and had really mind-blowing insights into what that integration with R would really look like. But by the time I became interested in working on this stuff, he had moved on to other things. And with his endorsement, I created a couple of sequence of pipeline tools of my own. First one was called Drake, which lasted a couple of years. But then realizing that I was hitting design constraints, moved on to Targets, which has been around for about five years or so, five or six years, and really pleased to see the adoption.
And there was also the mental model of a workflow, a mental model of a data analysis pipeline. And I think that more than the computation, more people in the statistics community get value out of having a mental model, a graphical mental model for what exactly is going on in a research project or simulation study or anything of that nature.
Using targets at Lilly and beyond
And internally at Lilly, we use it a lot for clinical trial simulation, really nuanced studies that we're trying to design to figure out what models we should use, what sample sizes the study should have, and things like that. Really diving into specific targeted questions that require a lot of custom code instead of tools that we just write and people can grab off the shelf, pull up a browser, pull up the Shiny app. The problems that we use Targets for are just just everyone is different. Everyone has its own complexities. And we also use it for model testing. So if I'm writing a Bayesian model and I want to run a simulation-based calibration checking workflow, that's a tool that I would use to architect the pipeline.
And externally, it's really interesting to see the uses that have come up that are off-label to, let's say, wrangle a pipeline of hundreds of Quarto reports or have a pipeline that ingests data every day and just publishes a dashboard or updates a dashboard on a daily cadence. And just seeing feedback from the community has really opened my eyes to what people use a pipeline tool for. And it's really versatile. And I think Targets is very opinionated, and it's interesting how useful it can be, even if not all of the opinions are universal among its users.
Right. Well, we love an opinionated framework here in the tidyverse world, although I'm a Bayside girlie. Don't tell anybody.
Well, I wanted to ask you selfishly really quickly. This is small. We have to get to sleight of questions. Because I learned SAS and R at the same time, and I was forced to learn SAS Enterprise Guide, which is like the point and click version of SAS, which I can't. Not my favorite. It has the process flow screen where you have your graphical overview of all of your processes, like webs and interconnections, and it's like nodes and edges. And I was wondering if you got any inspiration from that, because it's sort of a similar thing where you can go to one portion of it and rerun it, and it will only rerun downstream of that.
That's interesting, because I had no idea that SAS had a feature like that. I have the privilege of only being required to use SAS for one class my first year of grad school, and then I never had to touch it ever again. And I feel like I'm really lucky, because it's so ubiquitous at my workplace and in my field, not only statistics, but especially in pharma. And so I feel so oddly ignorant of everything and anything that SAS does. But I think that my inspiration came from Make, and from its underlying directed acyclic graph. At the time, I wasn't really aware of other pipeline tools like Arrow and Nextflow and things like that.
I've never used Make, Arrow, Nextflow, or any of those. So I have a lot of targets. FOMO, I've never used targets before either. Everybody loves it. There's so much love in the chat for not just targets, but for Draco for that. So from all of us, thank you for your service. You've done a lot more than I think that you realize, and we all love it. So let's get to Slido. I have a question from our very own Mike K. Smith. Mike, would you like to ask your question live? I'm going to stick it in the chat as well.
Balancing open source work with a day job
So let me. Hi, Will. And thanks again for all the great work that you're doing. I remember someone once saying to me that if you write an R package, you'll never be lonely because everyone will keep sending you support requests and comments and stuff. And so what I see from the outside is that you're doing great work to try to kind of separate out that development support stuff and your day-to-day job and personal time and life like that. How the heck do you do it?
That's a great question. And gosh, this is what made early career a particularly hard time for me. I'm sure just everybody starts their first job, they leave school, they join industry. I had all these ideas in my head that I just wanted to get out at the time. And I was trying to start a new job at the same time. And I compensated by effectively working two jobs. And that was hard. I didn't do it very skillfully in the first few years. I think that now, I think that a lot of things have improved that because I think the company's interests and my interests are more aligned. I had a lot to learn when I joined. And just like anyone, I mean, pharma is a huge and complicated industry. And for the problems it works on, it's critically important to get it right. And it's critically important to absorb everything I can. But as I transitioned to a more capabilities role and targets became more fundamental to our simulation infrastructure, those interests aligned and things naturally overlapped, saved a lot of time on my end. And I think that as far as responding to people
online, I think that there's some of the most fun conversations that I have on a daily basis, because you just dive right into the technically interesting details of a problem. So answering people's questions, although sometimes it's a struggle to find time, but it has been a lot of fun and I have learned a lot from it.
And I think lately I've noticed a bit of a slowdown in the throughput of questions that I've gotten about targets externally. And if I had to guess, it's probably a combination of the fact that targets is pretty mature right now. And also because people are getting a lot of answers from AI as well. So I think they're both of those things. Yes. I think support tickets and questions have just slowed down overall across the board, across all industries and sectors and walks of life because people are getting answers in different ways, which is a whole topic all on its own. Thank you so much for your question, Mike.
We have, let me see. I saw one from, sorry. Sometimes when you guys add questions to Slido, the question that I'm reading will be like, boop, and disappear and jump around.
Okay. I saw that Noor had asked a question and I would love it if Noor could ask that live. I am going to highlight it in Slido and I will also put it in the chat here.
I will. Nice to meet you. Hi, my name is Noor. Can everyone hear me? Excellent. So I'm about, for context, I'm a bioinformatician, but my background is in math and stats. But as someone who's primarily worked with tools like Snake, Make, and Nextflow, why should I use targets or what are some benefits you would say are using targets over something like Nextflow? I have a sneaky suspicion since lots of my tools are in R, I may see a benefit for using targets versus wrestling with Nextflow. But if you could get some more sort of like context or insight.
That's an excellent question. Really, at least something that I've been thinking about for a while. And although I've never used Nextflow or Snake, Make seriously, I think I've heard enough and read enough about it to say that the main thing that I'd go for, for deciding which tool to use is how integrated is your workflow with R. I think that with that, because targets says, there are hundreds of pipeline tools that are language agnostic and for these other purposes, targets makes a bet that if you really you really commit to R, then they're going to be just workflow advantages for people who spend most of their time and do most of their work there. And that's, I would say that that's really the overriding critical question of what tool to use in that sense.
Also, traditionally with, I've noticed people who use R also, if they need distributed computing, high-performance computing, then we often go more towards traditional schedulers like Slurm and we use Altair grid engine at Lilly and things like that. I've tried to get into the cloud distributed computing space through crew and its extensions there, but it's just never been as big a part of my day job because there isn't as much of a demand for those kinds of resources. So I guess it also depends on the computing environment as well.
Okay. But I would love to see if, I mean, anyone knows of or is interested in comparisons between and among targets, SnakeMake and NextFlow, especially the comparison with NextFlow, then that would really be interesting. NextFlow is just a completely different conceptual model. It's based on data flow programming instead of make-like programming on the same way, on a directed graph.
So Targets uses something called Khan's algorithm for topological sort, which is how it traverses its own graph with a little bit of a twist that you make dynamic updates to the graph sometimes for dynamic branching. And that's very different from NextFlow's model where edges in that graph are first-class citizens. And to understand sort of what problems and areas of flexibility it can handle and what kinds of efficiency gains you have there, that's something that I'd like to better understand. I smell a blog post in Noor's future.
Noor, you just got homework from Will Landau. Okay. We had another question and I believe it was asked with an asterisk, so I will read it. It was from Daniel. Daniel had asked how Gen AI has affected you as a package author and maintainer. Mostly from other people submitting issues or PRs and less about how you use it personally in your own work. But if you'd like to answer both of those, let's do it.
Gen AI's impact on open source development
Well, I think that as far as the daily impact of open source development and collaboration and things like that, the slowdown of throughput of issues and questions I think is the biggest impact that I've noticed. I don't often get pull requests relative to the number of issues that come my way, but I have heard reports that there's this challenge that a lot more, many more pull requests have been AI generated and to manage that and manage the communication with the human behind them I've heard is challenging for a lot of people.
I think even more of that though, I'm questioning the value proposition of being a package developer. Because writing code has never been an end result anyway. It's always ever been a means to an end. It's always been just a capability to solve the problem and it requires for a full value proposition as for a career, you have to have the domain knowledge and to know where to find those problems and apply them. Now, because code is so much cheaper than I'm thinking, in what ways do we have to reinvent ourselves in the AI age?
Even more of that though, I'm questioning the value proposition of being a package developer. Because writing code has never been an end result anyway. It's always ever been a means to an end.
I think that the thing right now is this transition, if there is one, is pretty slow because code is cheap, but good code is not cheap. Humans still very much have value, but I've been trying to transition a lot of my work to strategy and systems in addition to just being somebody who writes statistical models, implements models or implements simulation infrastructure and tools.
I actually just open sourced a guide that I'm working on, a model for developing agents that we're adopting internally to solve targeted problems. I'm calling it a paradigm of trusted mini agents. I know there's a lot of activity in the chat, but I'll put the guide here as well. It really dives into something that Sarah Altman said in her talk with Simon Couch at Gen AI Day this week, where she said that we recommend that it's important when you're designing an agent to design a scaffold so it matters less if LLMs are wrong.
What I've been trying to do for the last several months internally is to really take that idea as far as I can take it to its inevitable conclusions for targeted problems. Fortunately, just right before this hangout, I got permission to open source this guide. Huge thanks to Eric, who's on this call, who really facilitated this process quickly of getting it out there.
Yay, Eric. We love Eric. Everybody loves Eric. Hey, Eric. You can see him in your grid if you're looking because he has the coolest hex grid background ever. I always love it. I do.
Crew, mirai, and distributed computing
You mentioned crew earlier, parallelization. Speaking of doing multiple things at once and also enabling distributed computing across clusters and stuff like that, do you want to talk a little bit about integrating mirai to do things even faster? I wish Charlie Gao was here today. I don't think he is, but Charlie Gao is the author of mirai. He's wonderful. He was also on the data science lab with us recently to help explain mirai to us. It broke all of our brains just a little bit, but we got there.
Right. Crew came about because I wanted to separate out the distributed computing layer from targets. I wanted to come up with something very general that just had the kind of distributed computing that targets needed and that was versatile enough to work on a bunch of different clusters and cloud computing as well. There's a really important side of this, which is just there's a lot of variability from cluster to cluster. There's a reason to have a mechanism for inter-process communication that's very general and that doesn't depend on the specific configuration of each cluster. A really good assumption is to transact over the local network to send tasks to workers and receive them back. That required mirai, really.
Crew wouldn't have existed without mirai. mirai was just the perfect tool to transact across the network for that. I saw mirai just being discussed in the open source distributed computing infrastructure and R back in early 2023, and I started talking to Charlie that way. We've been collaborating on mirai and Crew for about three years. He's a really amazing guy.
At that point, he hadn't been very long in this space. He had an earlier career in investment banking. It's just mind-blowing not only how he transitioned into the space, but just how good he is at what he does and how insightful he is all the time. He's been super accommodating as far as collaborating on mirai. I bring this arguably very strange use case that a lot of people have a hard time wrapping their heads around. He's just been fantastic in solving this crux of a problem that really allowed targets to really fly on all these different platforms.
He's been super accommodating as far as collaborating on mirai. I bring this arguably very strange use case that a lot of people have a hard time wrapping their heads around. He's just been fantastic in solving this crux of a problem that really allowed targets to really fly on all these different platforms.
Awesome. Amazing. I wish Charlie was here so we could hear all of this wonderful stuff about him. There's a lot of low-level, interesting magic that goes on to send tasks to distributed workers and to manage how those tasks run and what to do when a task is finished and resolved and just to manage that communication and broker that communication. Wonderful. Thank you so much for giving us a little bit more info. I think everybody is going to go investigate the targets package after this call if they haven't already done it. It might see a little uptick in downloads and installs.
Communicating uncertainty
Well, we had an anonymous question that rose to the top in Slido, so I thought that I would ask it because I think it's a great general statistics question. It is, how do you communicate uncertainty to people who want a single number? Oh, that's interesting. What a great, broad question.
Yeah. I remember working cross-functionally on studies and studies and, and hearing from medical colleagues very confidently that, you know, we were looking at, we were looking at studies and, and hearing from medical colleagues very confidently that, you know, we were looking at, we were looking at, at sort of mean treatment effects from three different studies in the same master protocol, and there was a slight downward trend in them. And, you know, we were, what I was hearing from from medical was that they were very confident that, that there was a significant trend going a certain way in, in, in the responses and, and it's, you know, with the communication
is difficult, because it has to be done carefully enough to protect the relationship, but also communicate the fact that apparent trends like this could just happen by random chance, and how what that what that random chance looks like is very difficult to communicate to people. And I mean, so in visualizations, we produce, of course, we add uncertainty where we can, where the where the model justifies it. And, and to caution people against, against these kind of and against just looking at estimates and to always look at the and interpret the uncertainties as, as well. And and that's, that's always been just really important about just a piece of a statistician's toolkit when it comes to when, when it comes to communicating, especially, especially cross functionally. Because that that that happens really all the time.
All right, I wonder if that counts as our career question for the day. What I mean, I don't think that we're going to come back to that. But I would love it if we could ask the question that Natasha is famous for asking at this point, because I think it would give us a little bit more insight about your everyday. Natasha asks, what does an average day or week look like for you? How is your time divided between development, consulting, etc. And I would also love to hear from you, development, consulting, etc. And I would also throw in there, what tools do you use? Like, are you mostly in R? Do you use Python? Give us the details.
Yeah, so I think this varies a lot seasonally, as as major projects come and go. There, there are some things where it's an all hands on deck effort. Let's say if the team wants to use a fancy historical survival borrowing model that's never been done before. And we need to pull together with the with their team to spin up the simulation pipeline and have results by the end of by the end of the week or the end of the week after that. And that's that that's a lot of heads being put together at once. And so there are times when you know, all day, every day might be spent on, you know, helping with some with some simulation projects to, you know, for example, compare these new models and, and to compare them in terms of power and type one error across a variety of scenarios that might break the modeling assumptions or, or simulate model misspecification. In in quieter times, you know, we work on, we spend a lot of time working on general tools. And they're, they're, of course, you know, meetings to share learnings and things like that, both for our group and across the company. And there's, there's, you know, a lot of time is developing internal tools and capabilities meeting with small groups of about three or four people. And that that's the kind of cadence and, and stuff. And there's, there's a, you know, culture internally of seminar series and internally, both for statistics and for, and for tooling. And so those are those are helpful just to spread around knowledge about things that we're learning and conferences that we go to and, and, and, and the knowledge that we bring back from those. And, you know, it's really a mix of that and, and, and a personal focus time to implement things.
Perfect. Thank you. Thank you for the great question, Natasha.
Using targets for agentic LLM workflows
Well, I see a question from Tim Golden, Data Science Hangout speaker alumni. Tim, would you like to ask your question live? I'm going to highlight it in Slido in just a second. And I'm also going to stick it in the chat. Sure. Yeah. My question is, have you had feedback from people using targets for agentic LLM workflows for testing sort of parameterization changes of rag ingestion and associated changes to how the LLM instructions are dealing with that, or just anything in that sort of creating and testing agentic workflows space?
Oh, that's really interesting because that's, and I haven't, I haven't gotten questions like that specifically for, for LLM powered workflows, but I mean, targets tries to, tries to create a, a reasonably deterministic scenario in the sense that it assumes that any randomness is produced by, or apparent randomness is produced by random number generator, regenerator algorithms native to R and that it, it has really opinionated control over the seeds that over the set of parallel safe seeds that it uses for that computation. And if the temperature is anywhere above zero in those LLMs, then you're really going to get different results every time. And depending if it's, if part of a pipeline is an individual analysis, then, then it's, you know, there's, there's a different, a different set of expectations that you might have to have for a pipeline like that, but. I'm thinking of something like using vitals, the vitals package as an intermediary to determine whether you've got a binary pass fail for that particular area of the targets pipeline. Yeah. I think that it could be, if, if you're doing a vitals pipeline to just, for example, check, check and compare the performance of different versions of models across, you know, on a, on a set of, or multiple sets of test questions, then, then I mean, yeah, that certainly would fit in a targets pipeline. And, you know, occasionally I've talked to the, the tidy models team in general about, about potential integrations. And they, it seems like they have their own workflow solutions. Some of their, some of their stuff is compatible with, with targets in a sense that you might have a target in the pipeline for each workflow in, in the tidy models ecosystem. And there, there are some things that they like to do internally, like different parameter tuning algorithms and, and, and things like that. But, but yeah, I can see that's, that's an interesting question because there really is a lot of overlap and whether to rely on the tidy models workflow ecosystem versus targets as a tool that is responsible for a lot of the high level stuff in the workflow is really a, it's really a case by case project by project thing.
Oh yeah. You can keep going. Oh yeah. I was going to say where I've seen more integration with targets and tidy models is more of the classic the classic machine learning that isn't necessarily a neural networks. You think, you know, XGBoost or random forest models being together in pipelines. I've seen that more often than in the neural net LLM space.
Bayesian versus frequentist in clinical trials
Perfect. Sorry to cut you off. I didn't know you had more to do. But let's go to another question in Slido. This one's anonymous. And it is also a more broad one that I like. It says, when do you reach for a Bayesian approach over a frequentist one in a clinical trial? And how do you explain that choice to stakeholders? That's an excellent, excellent question. And I think the answer that most people are familiar with who, who think about these things is do we have prior information? How do we use that prior information? And do we want, do we want to be opinionated about incorporating that through a specific or rigorous prior distribution on the parameters? And do we want to have, you know, other techniques that may go beyond priors for historical borrowing? That's, that's, you know, those are those, that's, that's a use case that a lot of people are familiar with. But, but we've, we've implemented, I've certainly implemented my fair share of Bayesian models that, that run most of the time on diffuse priors.
And the reason is because decisions in early phase get a lot easier to quantify. And it's, it's, you know, easier, easier to, because we talk about in statistics, right? We talk about statistical significance. Is this result just the, just the result of random chance versus practical significance? Do we have enough evidence of observing a large enough effect size? And because of the flexibility in Bayesian models, it's much easier to, and much more convenient to answer that second question. You can define, let's, for example, you can, you can more easily quantify the posterior probability of observing a large enough, enough effect. And that's incredibly useful in early phase studies where, where we're taking the first look at, let's say the efficacy of a therapy and for internal decision-making, we want to say, do we have, do we have enough confidence in this effect size to move to a registration study? And that's a, that's a large phase three study with potentially thousands of patients. And so, and it turns out that not only do you not have to have a specific parameter in that model for, for the way that you're quantifying the treatment effect, you, you don't need frequentist asymptotics, like, like asymptotic approximations, like the Delta method to get at the information that you're trying to look for. You can use MCMC or Markov chain Monte Carlo and transformations on posterior samples without, without using a Jacobian adjustment. And that's, that alone adds just a lot of convenience.
And I guess I would also add to that, that there is, you know, I had a professor at Iowa State for a class who, who framed this really well. And in Bayesian statistics, we take a, a philosophically different view of probability. And, and it's, it's really useful just to, to use, probably to use it, to think of it as a quantification, a quantifying a way to quantify the degree of belief in something in general. So this is what's what my professor in grad school called the epistemic view of probability. It's a measure of how much we believe something, which is, which is more general and flexible and gets arguably more to the point than, than a frequentist definition of probability, which is just a long run ratio of, of repeated simulations of an outcome.
So this is what's what my professor in grad school called the epistemic view of probability. It's a measure of how much we believe something, which is, which is more general and flexible and gets arguably more to the point than, than a frequentist definition of probability, which is just a long run ratio of, of repeated simulations of an outcome.
All right. Fantastic. It's so funny that we're hearing MCMC, RAG, lots of acronyms today. Well, thank you for defining MCMC for us, for anybody who didn't catch RAG, that's Retrieval Augmented Generation. But I am creating a video right now about how defining acronyms when you're talking to people is part of data science community building because we use them so much. And oftentimes they're sort of niche, like not everybody in data science has used a Markov chain, Monte Carlo. So thank you for that. But speaking of that, Michael had a question, Michael Larson, if we could find Michael to get him unmuted to ask his question live, and I will also put it in the chat. Okay. So my question was, or is, what Bayesian MCMC tool do you use? Do you use STAN or the macro typewriters to get to STAN? And STAN is the software that uses the Hamiltonian algorithm to solve the MCMC question. I use BRMS because it's just easier for me to formulate the regression model.
What do you use? That's a great question. I think that STAN is a tool that I go to often for these problems, just because of how versatile it is. The advantage of Hamiltonian Monte Carlo is that as long as your model has continuous distributions for the parameters, then there's a whole lot of different models you can fit. And the syntax allows you to express it really easily because all you need to do is define the log joint density of the parameters in the data. And that creates a lot of opportunity. And it is a really fast, really powerful toolkit. I think that I sometimes have run into models that are just too big or complicated to fit efficiently in STAN. And occasionally, I've just implemented my own Gibbs sampler in C or C++. In fact, my grad school dissertation project, I was computing in CUDA, which is directly on top of an NVIDIA GPU. And I haven't done that since grad school, and I'm way out of touch with the technology there. But one tool that I've used for some frequentist modeling really easily is called RTMB. TMB stands for Template Model Builder. And if you're with the MMRM package, Mixed Models Repeated Measures, that's what it uses. There's a layer of R on top of it now called RTMB. And it's just magic because you just straightforwardly write the objective function in R. And you can plug it into after running it through RTMB for a translation step, you can run it in any optimizer. You can fit it with maximum likelihood or restricted maximum likelihood in really any optimizer in R.
And for Bayesians, there's this package called TMB STAN, where if you have an objective function that you've written in R, you can just like magic translate it into a Bayesian model that runs HMC or Hamiltonian Monte Carlo with STAN as the backend. It's not quite as fast as direct STAN code for big hierarchical models according to their journal statistical software paper, but it's, my gosh, that's going to be the first thing that I go to to prototype the next Bayesian model that I write. And it's really, and the integration with R is just magic. I mean, for those of you who use STAN out there, the syntax is kind of a cross between C++
and its own statistical helper functions and the going back and forth between, for debugging purposes and the compilation steps are a bit tricky, but just the ability to write out your model all in R and have it run just as well as if you had programmed it in pure STAN or anything, it's really amazing. I can't say enough good things about that toolkit. Amazing, wonderful. Thank you for the question, Michael.
It was obviously a very good one. Well, we have seven minutes left and I would like to get in one more question if we could, maybe two, because I'd love to ask a career advice question. So I see a smaller one from the bottom of the Slido. It's anonymous. I think I know who asked it because I saw it in the chat. It says, are you exploring AI agentic to automate clinical data analysis, ready clinical trial data and standard clinical trial data?
AI agents and trust in clinical trial workflows
As a function, we're exploring this as in-depth and as fast as possible. There's, I think, my group and I were more on the clinical design and modeling and simulation space than we are usually on the analysis deliverable space. But certainly we have a lot of visibility into that space and there's a lot of work that's being done there. And on top of that, there's a lot of everywhere, there's a lot of usage of things like cloud code and codex, just general coding agents to try to take a first crack at that problem. And what we're finding is there are a lot of things that can be prototyped very quickly and easily, but trust is the attribute that's just hardest to come by. And trust is the substance and the currency that's just become increasingly rare these days.
trust is the attribute that's just hardest to come by. And trust is the substance and the currency that's just become increasingly rare these days.
And for specific parts of that and similar problems, my group and I are really pushing this trusted mini-agent pattern to try to break down problems into ones where we can develop an agent that engineers hallucinations out of the system. Again, to Sarah's point from Gen AI Day, that we design the scaffold so it matters less if the LLM is wrong. And we're thinking about a lot of this in the simulation space where, as you'll see in the last chapter in that guide, that Quarto book that I linked to in the chat a while back, it's an example in trial simulation. The statistics in it are way oversimplified, but it gets to the concept where instead of the LLM producing the results or the LLM just independently figure out how it's going to solve this problem, the purpose of the LLM is relegated to a translation layer between the intent and the goals of the user who might say, hey, I want to simulate this trial under a couple different scenarios, one for power, one for type one error, and explore a couple different sample sizes. Translating that intent into formal simulation parameters that can be checked. And then registered agent tools run the computations in the background. And we have, from the engineering, confidence that results only come from those trusted tools.
So the question isn't anymore, do we trust these results? The question is, do we agree with the assumptions? And is this actually the workflow that the user intended to simulate? Is this design, as represented in these design parameters, what the user actually wanted to simulate? And I think that that can be accomplished through engineering and constraints without any prompt engineering. Because prompt engineering really is, according to the trust in many agent model, it's prompt engineering is about convenience, not correctness.
Career advice
All right. Well, can we get a 60-second career advice answer in from Will Landau at the top of the hour here? Is there a piece of career advice that you wish you knew before you started in data science or got into data science, or maybe even were in your PhD, or anything that you think would be helpful for everybody else here?
The best piece of career advice that I've gotten from my family is, and this was during my early career struggles, was don't run away from a bad thing, run towards a good thing. And this is really important for me because it was a culture shock going into industry. There was a lot of just soul searching. What is my place in this company? How do my interests and the company's interests align? And is this where I see myself? And those are really big questions. And I'm really happy in the role that I carved out for myself, and colleagues have helped carve out for me. It took a lot of work to get there, and it took sticking with it. And I think that I'm really glad that I did because there was just so much overwhelming positive stuff right where I was.
don't run away from a bad thing, run towards a good thing.
And everybody's decision about their career direction is different, but that's just a particular piece of advice that I got at one of the particularly difficult times in my early career. And I reflect on that often, and it's kept me going. Amazing. I think this is great advice. It falls right in sync with something that someone told me one time, which was Libby, comfort zones are great, but nothing grows there. It's kind of a barren wasteland, your comfort zone. I was like, really? I really like my comfort zone. All right, Will, this was fantastically fun. Thank you everybody for joining us. Will, I hope you had a good time. I did. Thank you for having me.
Amazing. Thank you everybody for joining us. As a reminder, the thing Rachel asked you to do was to go, if you wanted to fill out a G2 review, now that we've learned what G2 is together, if you have the bandwidth to go do that, go do that for her. That would be amazing. A big thank you to Rachel for being the reason that we're here. Also, closet comp tickets, you can still get them for the discounted price until midnight on Monday. So definitely go do that. Maybe like midnight Eastern time. Do it before Monday, if you're going to go in person to Houston, Texas this year. Thank you so much, everybody, for joining us. If you want to save the chat, top right corner of your chat, there are three dots and you can save it there so that you have all the resources that we've been sharing. We will see you next week. Please have a safe weekend. Bye, Will. Bye, everybody.
