Transcript#

This transcript was generated automatically and may contain errors.

Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.

Can't wait to see you there. I'm so excited to move on then and introduce our featured leaders today at the Hangout. We are talking to Kyle Austin, Principal Programmer Analyst at Thermo Fisher Scientific, and Martin Brown, Statistical Science Director at PPD. They both work at PPD, which is a part of Thermo Fisher Scientific, and we will get into what that means. PPD is the CRO, or Contract Research Organization, arm of Thermo Fisher Scientific.

So I would love, Kyle, if you could introduce yourself for us. Tell us a little bit about what you do and something you like to do for fun. I'll remind you of that part.

Oh, that's tempting. Yeah. Hi, I'm Kyle Austin. I'm a Principal Programmer Analyst. I have been doing, I've been in the clinical research game for about 15 years and 20 days. I just had, oh, 10 days, not even nine. Ooh, 15 years, my 15-year anniversary at PPD. I started there as an intern. I am now a principal. I've grown with, my career has grown alongside the company, which has been fantastic. I have a bachelor's degree in mathematics and statistics, and I was going to get one in computer science, but then I got offered this wonderful job that I have been in since I've been out of college. So this is what I've been doing for the last 15 years. I started off doing clinical programming. I have transitioned into a more school development role over the last five years, since, yeah, five, six years. So, but I originally did clinical trial data and analysis. I would do everything from every bit of things that I could possibly do, from data reports at the beginning to developing in STTM specs, STTMs, working on Atom, table listings, figures, all that good stuff. And I, like I said, I've transitioned now into a role where I am doing more tool development. I'm working more with the backend of developing tools for our users in-house as well as externally. Yeah, that's where I am now.

Wonderful. And we have two questions for you. What do you do for fun? And what is your background?

Oh, it's an IoSauron space. To be honest, I remember seeing it a while ago and I thought it was cool. And I put it in, I saved it on file. This might be 10 years old at this point. I can't remember the name of the galaxy, but I just thought it was really cool because it looks like the IoSauron. So obviously.

I think it's the protoplanetary ring around Fomalhaut. This is something that I Googled when I first met you and I saw this, I was like, this is the coolest thing ever. I put a link to a popular mechanics article about it. So everybody can go look. Yeah, it was, I just thought it was really cool. And it was, it was a, it was a file I had on here. I was like, that'd be fun for this call. So anyway, there's that. As far as for fun, I am a gigantic nerd across the board. I have almost 2000 comics. I collect those. I have a pull box. I read, I read everything, books and comic books. I play video games. I used to play a lot of D&D. I don't as much anymore, but I definitely, I have a board game closet behind me. You can't see beyond the IoSauron, but yeah, I am just your blanket nerd across the board.

Amazing. Well, you are in very good company because we all read comics, play video games and board games. So everybody here will be very excited to talk about all of that. Um, Ooh, um, well, yeah, I'm going to say, put that in the Slido, that question for Kyle. Yeah, I see some stuff pumping through. Good night. Love it.

Well, let's ask Martin. Martin, could you please introduce yourself? Tell us a little bit about what you do and then I'll remind you to let us know what you do for fun.

Yeah, totally. Um, thanks, uh, the opportunity to be here and chat to everybody. So yeah, my name is Martin Brown. I'm a statistical science director, um, based in the, in the UK and very similar to Kyle that, um, yeah, I started here as a graduate, so 18 sort of years ago and yeah, I haven't got rid of me yet, which is, which is great, which has been, we've seen it sort of grow and, and change over that, that time. Um, yeah, I'm a statistician. I did maths and statistics at the university and yeah, was looking, looking for work. And unfortunately this, this job came up, um, sort of at the last, last minute, really somebody dropped out. So I then, um, interviewed on the Thursday and started the following Monday, which was crazy quick. And yeah, over that time sort of worked and led projects, um, done, done review of stats out, but, um, learned a huge amount and sort of over the sort of more recent sort of past been part of statistical science where we support people internally, externally on, on more complex stats sort of issues. And then part of my role is then sort of involved looking after the stats side of, of software. So I've worked with a lot of different sort of software types and yeah, sort of worked my way finally sort of back to, back to R and got involved in, in that. So yeah, so, so we, we sort of thought, yeah, Carl can sort of give more of the programming side and then I'll give sort of more of the stats side, um, in that sense.

Amazing. And what is something you like to do for fun, Martin?

Yeah, so I love Lord of the Rings. So I can't sort of leave that out. Um, I've sort of read it, I've listened to it, I've watched it. So I've seen the stage show, so I could chat sort of ages about Lord of the Rings, but, um. Uh oh, uh oh, me too, Martin.

So we could just do that for the next hour. But, um, yeah, that'd be, um, for people to hate that, that would be awful. But, um. That's dangerous.

But otherwise, um, I really like running cause, um, yeah, it sort of gets me out of this office and now my screen. I did the London marathon a couple of weeks ago, um, which was amazing. So I'll briefly show my medal off. Congratulations. Yep. Camera on crew. We are all clapping. That's amazing. We have lots of runners in DSH. Dare I say, we have a DSH running club. Um, I am not in it. I'm in a walk club. Um, but Rachel runs marathons. Ken runs marathons. Who else runs marathons here? Who am I, who am I missing? Is it Kevin that I'm thinking of when I'm saying Ken?

I'm so excited. Oh, it's Sean. Sean, it's you. Marathon runner. Thank you, Rachel. Okay.

The hex poster and getting involved in R

Well, we had a question that I saw from the chat and I know that questions are going to be in Slido and we have a great one we're going to start with in just a second. Um, but I wanted to point out, Martin, the thing behind you, you have a very sweet poster covered with hexes. Um, can you tell us a little bit about that? Yeah, totally. So this is a poster I put together about two or three years ago and the idea really was to flag up really quickly how to get involved in R and open source in the pharmaceutical industry. So, um, I took, yeah, took the idea of hexagons, which are brilliant. Hexagons are best against if you haven't seen that sort of video. And yeah, the idea was just to give people, yeah, these are all the ways to either find materials or to get involved with different, different groups. So, um, yeah, there's a, there's an online version, which I can put the link into the, into the chats where you can then sort of click to those, those things. But yeah, generally the idea was to try and sort of show people, yeah, this is either where information is or how you can get involved.

What does a CRO do?

Amazing. I think we'll get to some of those, uh, pretty quickly as well, but I wanted to ask either you or Kyle, if you could answer Zach's question in the chat and Zach usually have as an asterisk next to his questions in, um, in Slido, Zach, are you willing to ask your question live today or able? I know Zach's usually in the office. Let's see. Oh, yep. I can. I knew you usually have asterisks, but I wanted to give you a shot at asking live if you could. Okay. So Zach asks, could you talk a bit about what a CRO does? And as a reminder for everybody, CRO was contract research organization. Um, Kyle, would you like to start?

Yeah. Yeah. So, oh man, where to start. So yeah, the, the purpose of a CRO is they are hired out by pharmaceutical companies to do the number crunching. It's the simplest way to put it, do the number crunching, do all the research, but to ensure that the statistical methods and everything are being applied, um, starts with something called the statistical analysis plan, which is a SAP goes from there. You build out your study data's analyzed, but what I always explain it like I'm five. The biggest thing I always told my kids is what do I do for a living? I make sure it's those. The funnest thing is it's, um, like when you watch commercials and you see, oh, you can see all those adverse events. They list off those lists are provided by people like me and Martin. There's other things we do as well, obviously, but the biggest things we have to make sure that they're, we, we get everything together and submit it to the FDA for approval. That is the biggest part of our job is taking drugs or any other thing. And it doesn't have to be a drug, but it could also be like different tools. Um, like, uh, Dexcom and things like that. Like, uh, I can't remember the word, the, the scale. That's okay. There you go. But it could continue. Glucose monitor is the term, but there's a more broad term of those devices, those devices, those kinds of things. Also, all those things have to be clinically approved. And so the purpose of CRO is that we are independently contracted by companies to do the research, analyze it, and then provide all of the statistical analysis at the end to decide, yes, this is safe and effective or the big things it's safety and efficacy. Um, depending on there's phases of clinical trials, which, oh man, like I said, it gets dicey, but there's phases of clinical trials, depending on what phase you're working on. It may be, are we looking for safety? We're looking for efficacy. Are we trying to take an existing drug that's already been approved? Are we trying to figure out, um, I know, uh, there's many drugs that have been approved for one thing and then they go, oh no, this actually worked a lot better for this. So we're going to start using for this, but we have to make sure that it's safe and effective, all those good things. So that is essentially what a CRO does. Martin, you can definitely expand on that. It's more of a statistical point, but. Yeah, just to say it's obviously a bit broader as well. So it's basically any sort of part of the work, um, that goes into a clinical trial.

Kyle's sort of gone through the biostats side of things in terms of being contracted to do the analysis and the design, but yeah it can go from anything from looking after the sites, looking after the safety, doing the data management pieces, writing the clinical study report. So anything to do with clinical trials can be sort of contracted out to us. So we sort of, and then the sort of big thing about being part of Thermo Fisher is actually we can sort of do everything end to end. So we can even give all of the medicines that are required for the trial, we can give all the equipment. So yeah, it's just helping where we where we don't sort of own the drugs ourselves, or the patents or those things were given the work of any type.

Types of data in clinical trials

Perfect. Thank you so much, Zach, for asking an amazing question to get us started. And I wanted to ask, just as a sort of like, broad context building question that we usually do to get a little bit more into the nitty gritty. Can you give us some examples of types of data that you are working with inside of your sort of day to day projects? Maybe an example of a project that's more encapsulated? I will leave that up to either of you. It looks like Martin has something on his mind.

I can say I can start. And this is the joy of this the CRO, rather than a farmer, for example, that essentially, we've got data going around all the time. And with any, any given day, you could be jumping between so many different studies, particularly sort of mine, where I'm sort of consulting more on, on the stats for these different studies that you could easily sort of jump between so many different projects and clients and disease areas. So yeah, so basically, in terms of data types, it will, there'll be the basic sets of data for any clinical trial, as Karl mentioned, sort of adverse events, lab data, demographics. So you get a standard set of data across all of those studies. But then depending on the indication and the trial area, then your, yeah, your other data sets might look different. So for asthma, for example, then you might be sort of looking at, yeah, how much people are breathing out versus something like sort of oncology, where, yeah, you're looking at tumour growth and sorts of bits and pieces.

So yeah, so we have, we have the great joy, really, of not being bored. So there'll always be sort of something different to take a look at and think, how best do we sort of analyse this particular data? How best do we design the studies to get the right data in? Karl, do you want to add anything? Yeah, yeah. And it definitely, I think Martin and I come at it from two different endpoints. It's interesting because Martin definitely is more statistical side. I'm more on the programming angle, which is, we definitely like, we'll work together in pieces. But it is interesting to say, you know, like, in particular, I work on dashboard development on two different dashboards that we have in-house, two different products we offer, one with Martin, one without Martin. But they use the same base of an STTM, which is standard transformation, standard data

transformation mapping, which is important. And it's a standard, so we can utilise that to make dashboards that we can utilise across all of our studies. And it's wonderful and great. But for instance, I was talking about continual glucose monitoring earlier, I helped build one of the dashboards we have for that, which is intense. It's a lot because it's multiple layers. We have this large lasagna plot we were building. I think we're utilising that application. We are utilising Spotfire. Another application Martin and I are currently building is, we're using R Shiny. So it's definitely seeing the two different build styles I've worked on, many different, I think at this point, I'm up to six or seven programming languages I work on within a given week. SAS, R, Phoebe.net. Goodnight, Iron Python, et cetera, SQL. It gets to a point where I think at this point in my career, I have so many tools in different languages, I definitely, syntax is fun. So I like to tell my kids, you know, languages are fun. I have a child who's looking to become a programmer and I'm always like, don't worry about memorising syntax, worry about knowing how the structure flows through that language. The advice I'll give to anybody who's into programming, it's more or less the same, but you always got to remember, do you start at zero or one for counting, which is a fun piece. It's always one of the things like zero or one, where are we starting for counting? Because that's the biggest thing. Also, how does your programming language round numbers? Because good luck. Are we rounding? Are we flooring?

Yeah. Well, we have a question that I saw that was not in Slido, but it was in the chat that made me laugh, because someone said lasagna plot? Did you say lasagna plot? I did say lasagna plot. What's a lasagna plot?

Oh, the lasagna plot is, oh man, they call it lasagna plot because it essentially looks like a, because I can't even say it's a four by four because it's not even, but essentially the data is layered in such a way that it literally looks like lasagna because you see layers like this, but the layers can go both in the X and the Y. So you have different interactions. I don't have it open. Unfortunately, I could show you, but unfortunately I don't have it open right now. So I can't really share that, but yeah, it is. It is a very unique plot. It's very complex. And I don't know that I could do it justice without showing it. It's one of those things that to see you go, oh yeah, that looks like a piece of lasagna, but there you go. Somebody in the chat needs to find us a lasagna plot.

Yeah. I was going to say, if you do a quick search, you can see it's mostly when, when colors, yeah, you have more sort of, yeah, many colors and you're thinking, yeah, this data's sort of changing. Yeah. Either on the sort of the Y or the X axis. So you just get a sense of, yeah, how it's, how it's moving.

Well, Dave says it sounds delicious. And I just saw that Adam, Adam did add that to the Slido. So thank you so much, Adam. Adam says new today. Welcome to the hangout, Adam. That's amazing. I will mark that when it's answered and we will hop over to Natasha's question. Natasha has an asterisk in her question, so I'm going to ask it and it is, what does a normal day look like for you and what software do you use? I think that's a great bit of context building. I know Kyle, you mentioned all kinds of programming languages, including SAS and R and Python. Could you each let us know what software you use and what tools you use to do your work with data, whether that is analyzing or publishing or storing data?

A normal day: tools and languages

Martin, you go first. Yeah, sure. Did you want to go more from the day angle or more from the coding sort of angle? I think starting with tools is probably the most helpful because that will let people know, I think, part of what a normal day looks like. Yeah. So as I briefly sort of mentioned, yeah, a lot of my sort of more recent time and yeah, probably across most of my career, really, that's yeah, as more of a statistician and thinking about things like either sample size calculations, which are sort of important calculations for how many patients you need in a trial, you sort of find quickly that there was lots of software that could do different jobs and then particularly with statistical design. So it's been a sort of heady mix of sort of programs that are already out there doing a specific job and learning those and how to use those versus, yeah, sort of just mostly using SAS for a long, long time until then I sort of moved more into this consultancy position where you're thinking, do I have the right thing to do what I want to do, the type of analysis that I want to do?

So there was an amazing sort of time about eight years ago where I had a project doing meta-analysis and a lot of the software at the time for sort of meta-analysis you had to then sort of pay for and trying to get it in-house and was quite difficult. But then I found that there was a package called the metaphor package, which does meta-analysis, which is amazing package and just meant I could sort of straight away sort of jump into that project and do meta-analysis using R. And that was sort of one example of pushing me away from using SAS sort of each day to then starting to sort of do more in R and think about, yeah, how do I sort of learn this? How do I use all the brilliant stuff that was coming out of the time? So over the last sort of eight years or so, it's been sort of, yeah, most days in R learning something new. And then, yeah, and then that's obviously then with AI coming in, it's having AI sort of open as well. So I'd say, yeah, I'd say the majority of my time now is in R and then, yeah, there are moments where I still need to go back to SAS to do things. Yeah. So that's, yeah, that's me.

So there was an amazing sort of time about eight years ago where I had a project doing meta-analysis and a lot of the software at the time for sort of meta-analysis you had to then sort of pay for and trying to get it in-house and was quite difficult. But then I found that there was a package called the metaphor package, which does meta-analysis, which is amazing package and just meant I could sort of straight away sort of jump into that project and do meta-analysis using R. And that was sort of one example of pushing me away from using SAS sort of each day to then starting to sort of do more in R.

Awesome. Do you do any reporting or publishing or sharing? I think Kyle does, but I'm not sure about you, Martin. In terms of? Like, do you create parameterized quarto reporting or anything, or is your end product, the analysis itself that you're like passing? Yeah. So this is the awesome thing about, we can sort of get into this in terms of trying to move our departments sort of more towards R and open source, which sort of an interesting thing about how would you sort of bring that more into processes and other things where they're mostly SAS-based. But in terms of my day-to-day and going back to that, if something comes up where somebody needs to do something for a client, nine times out of 10, I'm thinking, how can I do this in an R product of some sort? So that's either, yeah, individual analyses or quarto, R Shiny. It's basically just thinking, yeah, what can I do here that's going to really add value to that client? So recently they wanted an exploratory table in a particular project. So I was able to just recreate that very quickly as it was, as a static output in about a couple of hours. But then I was thinking, how do I then make that more interactive for that client? So then I sort of thought I can bring this into a quarto dashboard. I can deliver the quarto dashboard to them as a HTML file. So yeah, so basically, but this is the interesting about open source, isn't it? That after, I think over time, we'll then think, oh, what's out there that I can do that not necessarily even R, but something completely different. But in terms of my context of trying to bring the department towards open source, that's where I'm sort of thinking, go, yeah, I can do this thing in R and sort of hand it over to somebody else or start up a new process where doing that more and more. Yeah. So that's generally my sort of thoughts.

Awesome. Kyle, do you want to add anything? Yeah. Yeah. My day to day is definitely a lot different than Martin's. I support. I feel like I'm on at least like 10 or 15 distribution lists at this point. So like my day usually starts with going through emails and help desking. Like a good part of my day is running the engine as we like to call it, but it's the reaching out to folks that are having issues with a certain tool we have in-house, going hey this isn't functioning or this something's going wrong here, can you look into it? And as a tool owner it's my responsibility to help folks with that, you know, whether it is engaging them directly, is it hey you're doing something, hey let's follow this, you make sure you do it like this, or is it an issue of the tool itself? So it's definitely, like I said, I have to, I wear many hats, but that is a lot of my day is that also I do tool development, but as far as like doing reports and things, I do internal reports for our teams. I've surfaced one of those in a Power BI dashboard, it's the other language that I've learned. So like I said, I've experienced Power BI, Spotfire, and now R Shiny. So I'm definitely learning and learning the different functionalities these tools have and the differences between them of like, you know, like this is great for this, this is great for this, and being able to know that. But I think that's part of my job too is knowing the contextualization of because I have experience in so many languages at this point, and knowing okay I'm doing this here, this here.

So right now we have our first in-house SAS R hybrid macro that I've built, where it is a SAS macro, and I had to figure out okay cool, I need SAS for this portion to handle libraries and do a bunch of stuff with that. Then I have to hand it over to R because I need R to create these parquet files that I need, and in the SAS solution we have currently we can't do that. So it's like okay cool, but I could do it in R via open source, which is another reason love R, it's open source so I can go out and grab a package and we can figure out okay cool, this package is what I need it to, but it's just even just the handover between those two languages, I was literally developing that and then passing all that into Snowflake, so I had another layer of complexity there, and that was a fun build. It's functioning now, it's running daily for like hundreds of studies, it's awesome, but it was one of those things, it's difficult when I'm dealing with things like that, and it's like okay cool, this is in SAS, this is in R, this has to go to Snowflake, how do these pieces connect, where do these pieces go. And I saw somebody say that they struggle with the back and forth between languages, and it is hard. I will say sometimes it is, I have to go okay cool, and I do it all the time, where like I'll forget to put my semicolons at the end of a SAS line and go oh duh, that's why your code failed. Yeah, we used to joke that SAS just meant simply a semicolon. It's right, yeah, so it's those kinds of things, but yeah it's good, but and I will say that things like chat GPT are extremely helpful these days, more than just a standard Google search to kind of figure out okay, I need to do this, and this needs to go here, how can I connect those pieces, and it definitely keeps my brain straight, especially like I said, but even then it's not the best in the world at everything. It definitely struggles with when you're dealing with three different systems like that, it was definitely chat GPT would feed me something like that's not gonna work, chat GPT, you can't use those single quotes and pass a macro variable in SAS that won't function. So anyway, but yeah, lots of things. Yeah, I feel like SAS is the least documented, at least publicly, language that it can learn from. So like if you're working in Python, you have a bunch of documentation, R maybe slightly less, and then SAS even less, because a lot of it's behind a wall.

Making the case for open source

But we had a question that was a kind of a nice follow-on to this that I wanted to grab. Let me go see if I can find it. Sometimes the Slido moves around a little bit as things get uploaded. It was an anonymous question, and it said how do you make the case that moving to open source is positive if the outputs are similar? If you're talking about well this one's in SAS, this one's in R, how do you make the case to either your clients or anybody above you that you should be working in an open source language or framework?

Yep, it's a great, great question. And yeah, there's a number of ways to definitely look at it, depending on, yeah, I suppose depending on your industry and other bits and pieces. And I think, yeah, I always think the big sort of positive really is having more, isn't it? I always think that's the sort of thing, really. SAS is great, and SAS has a load of great parts to it that, yeah, for a long, long time you can see that a lot's gone into it, particularly in terms of statistical procs. It's very solid and really, really good. So yeah, partly, yeah, but there are going to be things and ways that the SAS isn't going to do so well that other things and newer coding and things are going to do better and be more flexible about. So definitely as a CRO, definitely taking on different types of projects and as things seem to get wider and go out in different directions, that's where being multilingual is totally important. And I've seen in cases where, yeah, I've created things that are far better in R than in SAS, and then other cases where, yeah, it could be the other way around and SAS could be the best thing to use.

So I think if, yeah, if you're looking at it really narrowly and you're looking at just sort of specific static tabular outputs, for example, then maybe there's not so much difference, but that's not where we're going to be, but it's not where we are now where, yeah, the sorts of ways of digesting data is getting bigger and bigger. So we need the tools to use in different situations to be able to pivot, to be able to understand and take things. Yes, hopefully that sort of helps a little bit, but it's obviously, yeah, it's a big, it's a big ongoing conversation as particularly in pharma, as certain sort of companies will do things certain ways and then others will do things other ways.

So we need the tools to use in different situations to be able to pivot, to be able to understand and take things.

Awesome. Thank you. I like your having a sort of a director level view of it. I think a lot of people in the Data Science Hangout have experienced the transition where they're on

one side or the other of a transition. So you are in good company here. And I did see that David Onder had asked a very, very similar question. So David, thank you so much for asking that as well.

And I wanted to hop back over to Slido and we had a question that has risen to the top and Natasha had also added a reply to it to sort of flesh it out. The question is, as someone who is a statistician in more of a healthcare focus and looking to pivot into a CRO or pharma, are there any recommendations for upskilling? And Natasha added, is having a background in bio or pharma critical? Who would like to take this one? Who's thinking here? I can answer that. To get into statistical programming, you don't necessarily have to have a background in pharma. You don't.

Was it statistics or pharma we were specifically? Sorry.

It was actually coming from healthcare and moving from healthcare in a statistician role into a CRO or pharma. I think a lot of the same skill sets would comparatively. We're talking about programming, then yeah, I still think it's kind of the same.

It's, there are definitely differences. I obviously have always lived in this world and known this world. And I know Martin's similar with me where we've just kind of just, we've always been in this world. I've never transitioned over from another place, but I definitely know that from the programming side of it, it's not that bad, I don't think. Because I mean, realistically, this is my opinion, because obviously I started as an intern here and I was brought up in the company. I was taught the company ways. I learned CROs from a CRO. So I think that definitely helped me knowing that, okay, cool. I'm figuring this out. I'm learning pieces as I go. And I definitely didn't, I didn't even know this was a job when I got hired on as an intern. This is a job. You can just program SAS code for a living. I was like, that's cool. I always liked SAS in school. Like I said, I was a mathematics and statistics major. So I was like, oh, that's neat. And I always like computer science. I was like, oh, that sounds like a fun thing to do. And I genuinely enjoy it. Like it's, I like programming as a career. It's been awesome. It's definitely busy. It's stressful, like any job. There's definitely that. It's high stress. But I think that's also, I don't know, depending on who you are, I function under pressure well. So I make diamonds when I'm pushed. So I don't have that issue. But yeah, no, I may have wandered off there a little bit, Martin, if you want to put me back on course. No, no, it's good. I think the main point is that, yeah, there should be no concern about moving across from somewhere else. Because as we've been saying, like the skills, yeah, are very much the same, really. That being a good statistician, particularly, it shouldn't really matter what your data is. It's just having, and that's, this is important for AI, isn't it? Particularly sort of knowing what the key aspects of being a good statistician is all about. And being able to look at your data, get into it, understand the context, treat it properly, do all the normal stuff in terms of getting to know it initially first, and then sort of moving into sort of planning your analysis and all those sorts of things.

But yeah, I suppose in terms of upskilling, I suppose the key thing is just making sure you sort of understand more about the pharma industry, understand more about clinical trials, and the phases and how it all works. I think that's far easier now than it was when I first interviewed and they gave me a little pamphlet sort of telling me about things like, yeah, it's so easy to sort of go out there and see all about clinical trials. And yeah, they're getting more complex sort of all the time. So having a good sort of working knowledge.

But I think in general, like we've been saying, like the key thing is being sort of willing to find the time to start getting into things like coding and using AI properly and all these things. And a lot of these things you can do completely on your own, which is amazing. You can yeah, download sort of RStudio and simulate some data and sort of get going. So yeah, use all these things that are there for you. And I think if you go into an interview and say, I've been doing this sort of on my own time and people are going to be impressed, you're going to be like, oh, you know what sort of goes into creating an application, etc.

yeah, download sort of RStudio and simulate some data and sort of get going. So yeah, use all these things that are there for you. And I think if you go into an interview and say, I've been doing this sort of on my own time and people are going to be impressed, you're going to be like, oh, you know what sort of goes into creating an application, etc.

So looks like Kyle is nodding his head. Yeah. So I think that sounds like a good piece of advice. Well, there's a question in Slido that I think will help flesh this out and it's from Sean Highland.

I highlighted it in Slido so Sean can see it. Sean, are you able to ask that live? I will put it in the chat as well in case you need a reminder of what you asked. Okay, yes I am. So my question is, I know PPD works with a lot of different clients. How, from like a governance perspective or an application, do you keep different clients' data separated if there's different methods, tools, techniques that the client asked for, making sure that the results stay with that client only? Or do you actually take some of that information and see how that particular technique worked for that client and you're able to then use that as a new tool in your multi-tool to help out other clients in the work that they're doing?

Data governance and knowledge sharing

Yeah. Kyle, do you want to start from the governance process and then I can sort of talk more to the stats methods across things? Yeah, absolutely. So yeah, from a governance standpoint, our system is locked down. We have a very, very, very strict policy. Obviously, there's blind and unblind people who don't know. Blinded data is you have no idea who's on what treatment. Unblinded data is we actually know who's on treatment and that alone is a very like totally separate silos, storage areas locked down based on access. So for example, if I'm a lead on a study, I'm only going to have access to this area and I can grant access to folks who have access to certain study areas. We have an independent system that we have, our BTI is what we call it, but it's where all of the data is stored locally, right? Where all the data comes in, it gets imported, exported, all of our coding is done in there, but we do it by client, by study. Every single area is locked down. So it is very much a, there's no way that I'm accidentally going to get access to an area I'm not supposed to. It's a very, very, very strategic system. We use that throughout most of our methodologies to funnel that access through to make sure that we're not allowing folks access.

Unblinded access is handled in a completely different way. There's something called a URAF, which is a form that specifically states these 10 people are allowed to have access to this data. And only those 10 people will be allowed to have access to the data. So it's, from a governance standpoint, that's very easy. We do that. We just lock our data down. It's strategically done in that way to ensure, like you said, we accidentally don't put someone else's data somewhere else. We have protocols in place that God forbid, something does happen to accident on the blind. Is there a thing that happened in our industry where let's say, oh, the vendor screwed up and sent an unblinded data to the blinded person. They loaded it in. We have processes to go out there and destroy all that, lock it all down. It never gets beyond where it needs to be, et cetera. That has been something we've had in place for decades at this point. So yeah, we don't worry too much about that. And as far as sharing things go, that definitely happens a lot because we'll have, we have sister studies, things like that under one client we may be doing. And depending on a client, they have specific ways they want to do things. We may have certain macros we built in-house that, okay, this client likes their outputs to look a certain way. We handle that internally. And it may be that after working with a certain client for a long time, some of them like it some ways and some of them are smaller clients and like, we'll do whatever you want. We know they're open to more things like that. So it kind of just depends. But yeah, that's from the governance. I'll let Martin pick up from the stats side of it.

Yeah. So then the interesting thing is, yeah, how do you become more efficient and how do you take sort of learnings from study to study and yeah, make sure that the quality is sort of high on everything you do. So I think generally the way we've sort of approached these things is that, yeah, we'll have sort of expertise in particular areas. So then you sort of know who to go to and you're able to share sort of knowledge well. So as I mentioned, I'm part of stats science. So generally a lot of our knowledge is then on the more complex things. So if you've got issues with missing data or particular more complex side of survival analysis, or designs, then you know who to go to, you know how to get that resource. And then sort of programmatically, it's then sort of thinking, yeah, we've seen this on a number of studies. Yeah. How best do we then take that same sort of analysis and do it the same? But I suppose the difficulty is that things are oddly a little bit different. So then each of the people within our department needs to be better at sort of jumping from thing to thing and sort of knowing where to change things and where to keep things, keep things the same. So I'd sort of say, yeah, the more you can have that brilliant sort of knowledge and learning sort of within the people, then it tends to mean then you're covering all of the different studies really well. But yeah, we've definitely tried out lots of different things in terms of trying to, yeah, make our coding better and yeah, sort of applying similar macros or processes or packages now. So that's the joy of packages, isn't it? That you have a great package for a particular analysis, and then you take that, you're able to digest it and understand it and use it properly.

Data quality across the clinical trial process

Awesome. Thank you, Martin and Kyle for that. And thank you, Sean, for the great question. I see a question from Nellie in the Slido, which I have highlighted to put at the top. And I'm going to put it in the chat as well. I like this question because it covers another anonymous question that was in the Slido as well. So I'll let Nellie ask her slide. Hey, Nellie. Hey, Nellie.

Nellie, thanks so much for joining. I was curious if y'all have worked on improving the data quality of the data from the source of those who are actually inputting it, not just on the analysis side. And if so, what kind of actions have you taken?

Yeah, I can speak to that. You go ahead, Martin.

Yeah, I'll just briefly start, I won't go into too much detail. Yeah, so the amazing thing about data quality is that, yeah, it's a hugely great sort of area to really sort of improve all the way through the entire clinical trial process. So you can come at it right at the beginning and sort of think to really sort of improve data quality, then your design has got to be good, instructions, the clinical team carrying out things properly. And yeah, I'll sort of talk about that sort of area because it's an interesting area from a statistical point of view, because as the data comes in, you can do lots of exploratory analysis to see are there issues here. And part of that then is thinking, what can I sort of change now and just improve from a data management perspective? But even sort of then thinking forward, can I improve how the sites are doing things? So this is a sort of an amazing area in clinical monitoring where trying to do, trying to look at your data, trying to understand the quality, sort of from all the way from sort of detecting fraud, if fraud is occurring, all the way to just improving your processes as the trial sort of goes on. So yeah, this is an area that, yeah, I've been doing a lot sort of in, and we're just trying to sort of think, can we do sort of more analysis? Can we, before we get to the end where we can't make those, those changes anymore. But Carly, I'll let you sort of talk more to the, yeah, whatever you're going to talk to.

Yeah. Yeah. Yeah. No, no, no. Yeah. And it's, I'm coming from more of the other side. Like I said, when I started as an intern, I used to do DM reports, which is like the basic thing of like data comes in, we're writing out reports, which is surfacing things. I mean, it's anything from checking for, are there any UTF-8 characters in this data? Is that going to be problematic? Is there something in that data that's coming through that's going to break something down the line, et cetera, those kinds of things. And those things are deep. I mean, I, there's, I used to work on things called missing pages reports where ECRFs are something that's created where we have this major thing. You make these missing pages reports and it's going, Hey, these people didn't finish out, didn't fill out these certain forms. You know, we're missing this data from these people at these visits, that kind of stuff. There is a ton of early detection, as Martin was talking about earlier. So there's early detection. There's later on detection. I know one of the dashboards I work on, we service STTM and that data is looked at by our medical monitors. And then they go back and they'll review things like, okay, cool. We're seeing this, we're seeing patterns here. We're seeing an issue with this site. We have to go back and retrain. So there's, there's definitely lots of tools that we've developed in house that we utilize across all phases. And I think that's the difficult part is you can't just do it on the raw imports. It also has to come further down the line when you're actually looking at, when you're, you know, taking your STTM, which is the next step after as it goes imports, extraction data to STTM, to analysis database sets, to your tables, listings, figures, and everywhere along those lines, there are QC checks that are in place. And it's difficult to pinpoint what is the best place

because really it has to be across the whole piece because to Martin's point, you know, when you get into the analysis and you start looking at the data, start looking at the analysis and go, oh no, we didn't consider this when we set the study up. And that's something else I've noticed too, is that now on study startup, a lot more of the team is involved when they're building out those CRFs, which I think is fantastic. I know I've been personally on studies at startup where it typically wasn't where the programmers and the statisticians were part of that conversation. And now they are, which is crucial, I think, because it helps make sure that from the offset, you're setting these studies up to with the knowledge base of everybody going, yeah, cool. Nobody would have thought about this, but a statistician, nobody's seen that, but a clinical programmer. And now they're, they're part of the conversation, not just, you know, building it. So then it's from the beginning, we know, okay, this is great.

Yeah, there's, there's a lot. It is complex to the point of, like I said, there's so many pieces there to do, but I would recommend, you know, just taking at it from the best angle you can, trying to keep everything clean. It's difficult. It's an ongoing process. It really is. Clearly a big problem. And I'm seeing lots of nodding heads. Pedro is nodding like so vigorously.

Pedro and Megan and Nellie, everyone's nodding. So I think that you have some agreement there that it's a gnarly problem that deserves focus. Well, I primed Noor in the chat that her question is coming up. I primed her because we only have eight minutes left. So I wanted to let her know to get ready to unmute.

Nour, could you ask your question live? I highlighted it in Slido and put it in the chat as well. You didn't want me to ask the board meeting question. I'm so disappointed, Libby.

We only have so much time. But actual question. Teal, I hear a lot about it, especially in our medicine and like our Pharma realms. What is it? Why is it? Why should we use it? That's it.

What is Teal?

Yeah, great question. Teal is brilliant. Yeah, when I was sort of getting into R and trying to sort of learn it more, I'd sort of heard about Shiny and how brilliant it would be in the space of creating all of our outputs. So a lot with R outputs, you sort of create one type of table, and then you often then create a million repeats for different reasons. Say you want a different subgroup with the same table, or you want to look at serious adverse events rather than adverse events, so you end up sort of recreating lots and lots of repeats and the number of tables, listing the figures you create for a study, then becomes really big. So with Shiny dashboards, it was that real thought that we can go interactive with this, and then we don't need to sort of create so many, so many sort of outputs and it becomes interactive. So I spent time sort of trying to create my own dashboards doing this and got relatively sort of far with it.

But the joy was that sort of Roche at the time had been developing the Teal package and ecosystem to do sort of Shiny programming and doing it from a pharma perspective. So being able to use this ecosystem to then create particular dashboards that, yeah, create our clinical outputs in different ways, in a modular system, being able to pull in data, being able to filter by it, all of these sorts of things. So a lot of the work then that I was sort of doing and trying to think how do I then create my app to do it for millions of different studies, Teal had sorted out sort of for me. So it started out from Roche and it's become a great example of a cross pharma, yeah, sort of package and sets of packages. So it sort of takes a lot of the work out from somebody wanting to get started. So if you wanted to just create a very simple application from scratch, you can, yeah, quickly loading clinical data and then you can sort of take some of the modules which are rebuilt and then quickly create sort of simple summaries, simple figures, but then it's also flexible enough that you can then use the ecosystem to create your own modules.

but then it's also flexible enough that you can then use the ecosystem to create your own modules.

So that's what Carl and I are sort of doing right at the moment for sort of creating a particular dashboard out of the Teal ecosystem and looking to sort of share that with clients.

Thank you, Martin. And I will plug my friend Dror, who is giving a talk on this very topic today at R Medicine, is giving a talk called Not Invented Here, Balancing Unique Software Needs Versus Out-of-the-Box Almost Perfect Generic Solution, which is Teal. And I think part of his pitch is like, how much time are you going to waste building a bespoke thing if you think that your needs are super, super niche? Are they really? Probably not. Are they probably covered by Teal? Can you get 99% of the way with Teal? Yes. And you can do it much more quickly. And then it's a little bit more standardized and it's something that's proven to work well for your use case.

So catch Dror out. I think his talk has probably already happened today. You can catch it on the replay probably. All right. Well, we have four minutes left and I would really love to get in a quick, very quick career advice question. Kyle, do you have a 30-second piece of career advice that you would give somebody after your many, many years doing this? And then we'll move over to Martin.

Career advice

R. If you want to get into this industry now, learn R. SAS will be helpful, but I'm just saying. And I know because when I was in school 14 years ago, it's been a long time since I've been out of college. But when I was in school, R was for academics. So I ignored it. I focused on SAS path because that was the career. Now, mind you, knowing both is great. And I think it's just learning a language in general. I'm all about... I push this on my kids too. My nephew just is graduating with a computer science degree. I was like, learn R, learn SAS. They're niche languages, but man, if you can do them, there's careers there. So it's that. That's my recommendation to everybody. Learn R, learn Python, learn SAS for the empathetic value of being able to work with other people who know SAS as well.

All right, Martin, we have two minutes left. Do you have a 30-second piece of career advice that you could give us? Yeah. I think, yeah, get excited about data and hopefully everybody on this call is excited about data and that's why they're sort of here. We are, I promise. It's sort of the best time to be either a statistician or a data scientist or a programmer. It's an amazing time to be alive in terms of what's available to us. So it's really just throwing yourself into that. So it's really trying to think, what am I interested by? What part of that really excites me? And throw yourself into it. Ask questions, come to things like this. Just give yourself projects. So I think, yeah, some project that you're excited to do, if you had to do it on your own time or otherwise, yeah, just throw yourself into that and be sort of taken along with the ride and you'll meet amazing people and yeah, you'll have fun.

Ask questions, come to things like this. Just give yourself projects. So I think, yeah, some project that you're excited to do, if you had to do it on your own time or otherwise, yeah, just throw yourself into that and be sort of taken along with the ride and you'll meet amazing people and yeah, you'll have fun.

Thank you, Martin. That was a wonderful answer. Follow your curiosity and your excitement. Well, I know Rachel had a question. We have one minute left. Rachel, you want to ask that? Yeah, I just had a quick question because I heard you mention using Posit and Snowflake together and I just love when we can get Hangout people together in real life as well. So I was wondering if your team was going to be at the Snowflake Summit the first week of June in San Francisco, if anyone from your team would be there or if anyone at the Hangout will be there? Anybody? Anybody? I will not be there.

Yeah, not on my mission. They will be at home working, but that's okay.

Just checking. I heard our team talking about it today. So I'm in the office in Boston today and our marketing team's here. So I heard about some of the things they're doing at Snowflake Summit. I personally won't be there, but that would be a fun place for people to get together.

If you are out there and you're listening and you are going to Snowflake Summit, hop on the Discord and let us know. We can connect you with some folks to say hi to at Posit. Okay, we have reached the end. We are done. Martin and Kyle, thank you. We have one more quick thing for Hangout people who are in Boston as well. We are going to do a lunch to get people together at the Boston office on May 20th. I was super inspired by Isabella getting people from the Hangout together in Chicago. So I just want to make sure that everybody knows that. So if you are in the Boston area and want to join, if you just want to email me, just put my email there or LinkedIn or Discord. There's so many ways we all talk to each other. There are so many ways. You can tag Rachel or I or Isabella. Let us know. You can post in the Data Science Hangout or Data Science Lab channels. Those are good. You can post in the general channel anywhere. And Hubert, who is my hero, said he thinks that Dror's talk about Teal actually starts in 15 minutes. So if you are registered for R Medicine, go catch that for me because I'm not registered and I'm missing it and I have so much FOMO. Okay, everybody, thank you for hanging out with us. Kyle, Martin, thank you so much. This was wonderful. I hope you had a good time. I did. Thanks so much. Wonderful. Thank you so much. See y'all later. Bye, everybody. Thank you.