Resources
Jan 11, 2023 |
1h 4min
|
13.1k views

Shifting to an Open-Source Backbone in Clinical Trials with Roche

To hear more about how Roche and other major pharmaceutical companies are transitioning to open source data science you can watch talks from this year’s R in Pharma conference: https://www.youtube.com/@RinPharma/playlists

In the Roche presentation you'll hear a few specific pieces of tooling mentioned, including:
⬢ The pharmaverse - a series of packages designed specifically for clinical workflows: https://posit.co/blog/pharmaverse-packages-for-clinical-reporting-workflows)
⬢ Posit Professional Team Products like Posit Connect - a publishing platform for Life Sciences teams that need to deploy, collaborate on, and automate clinical reporting and data science tasks. (https://posit.co/products/enterprise/connect)

If your team is transitioning to R for clinical trials and looking for training, Posit Academy is currently accepting cohorts for organizations that want to teach their workforce leading open-source languages like R and Python. Our team has worked with many pharmas and we offer specific curriculums and expertise for statistical programmers and data scientists working in pharma. (https://posit.co/products/enterprise/academy/)

At Posit, we have a dedicated Pharma team to help organizations migrate and utilize open source for drug development. To learn more about our support for life sciences, please see our dedicated Pharma page where you can book a call with our team. (https://posit.co/solutions/pharma)

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

I'm Phil Bauscher, the Director of Life Sciences and Healthcare here at Posit. I'm going to help moderate the session today. We're going to be doing some Q&A. You'll see that popping up on the right-hand side. We'll also be doing Q&A through Slido. We'll share that link. We'll also be doing questions coming in from YouTube. So lots of exciting questions coming in, and we'll tackle those at the end of the session today over the last 10 to 15 minutes.

I'm very excited to bring to you today the eighth installment of our Life Sciences Series. Previously, we highlighted BMS, Merck, GSK, Novartis, Johnson & Johnson, and others. You can find these Posit webinars linked below from my team.

Today, we have an exciting sequel to our July 2019 webinar by Roche and Genentech that highlighted the role of R in drug discovery, research, and development. Today, we're going to highlight the awesome work being done by Roche and Genentech in shifting to an open-source backbone in clinical trials. This amazing work is being led by James Black, the Senior Director of Insights Engineering and Data and Statistical Sciences. We'll kick off the presentation today with Karen Martin, the R Enablement Lead and Statistical Programmer. We'll then pass it over to Thomas Niemann, Code Collaboration Lead and Data Scientist, as well as the Technical Lead for the Admiral R Package.

We'll then end with Ning Ling, People and Product Lead in Product Development and Data Sciences. Roche and Genentech is such an inspiration, from the legacy with Robert Gentleman to the contribution in countless hours to the open-source drug development community, a true lodestar in our space working with other pharmas and contributing to efforts such as Admiral in the Pharmaverse, the MMRN Package, the FDA Pilot Submissions, Bioconductor, R-Core, R-Tables, and countless other projects for the community. Over 10 people contributed workshops or talks at the R in Pharma Conference in November, and you can find those videos on the R in Pharma YouTube page linked below.

And it's with great honor and pleasure to bring this session to the community today. And now I'll pass it over to Karen to kick off the webinar.

Thanks, Phil. So, are my slides out? I'll just wait a moment.

Okay, great. Yeah, so thanks so much, Phil, for that intro. It's really great that our efforts are being seen in the wider community. So today, yeah, myself, Thomas, and Ning are going to talk about the next steps we're making at Roche, the steps we've kind of been making already and what's happening this year.

So I'm going to start by just providing an introduction as to what it is exactly we're talking about in terms of shifting to an open-source backbone and kind of why we've got to this point, why we've made the decision right now to switch there. And then Thomas is going to get into what that looks like in terms of the platform we're building to do our work on and the tools that we're going to use on that platform. Then he's going to hand back to me, and I'm going to talk about how we're getting our talent base ready to switch from one language to another, and also how we're kind of trying to move to a different culture in how we approach data science in general. And then we're going to finish with Ning, who's going to go into a bit more detail about how this business transition is actually going to look like in practice.

Setting the scene

So let me start by just setting the scene a little bit for what I'm talking about. Phil kind of alluded to this already, but Roche is obviously a really big company, and today we're not here talking for the whole of Roche. All of us are from a particular department at Roche called PD Data Sciences. If you don't know what that is, that's fine. I'll explain it a little bit now. Our basic job is to report clinical trials.

So Roche obviously conducts loads of different clinical trials all over the world to gather data on the effectiveness of the different products we create to help fight diseases. And our role in PD Data Sciences is to make sure all that data is gathered accurately, following the protocols we've set out, and to summarize it so that we have a good picture of how these drugs work. So as you might imagine, this is a really important job. It's really vital that we get this right, that we have the safety and efficacy data to be accurate, and that we can share it with regulators as soon as possible. Because the more quickly we get this data summarized and presented, the quicker we can get drugs to patients to help them fight illnesses.

The goal: a language-agnostic framework

So what's our goal then? So fundamentally, our goal in PD Data Sciences is to shift towards a language-agnostic framework. And what I mean by that is we're moving away from having one solution for how we do our programming for clinical trials. We want the ability to be flexible and move towards the best language for solving our problems. And our expectation is that in future, there won't be necessarily one solution for solving it. We'll be picking the best problem from the marketplace.

So most of what we're talking about today is the first step on this journey, but we will be alluding to the wider ideas of this as well. And that first step, I think, has already been mentioned a bit in the advertising for this seminar, which is that this year, from quarter two, novel studies will be using R as their core data science tool. So we'll be analyzing our data using R primarily.

I will say that this is definitely novel studies. Like, we're not doing a cliff-edge thing where we switch everything to R overnight, but it is a big change. We expect that over the next few years, we'll have more and more studies using R as their core data science tool until we're using it as our primary tool across all of our studies.

And this, as I say, is the first step in this journey where in future, we can move flexibly in the future. So R isn't our destination. It's part of a journey that allows us to move towards what the best data science tool is in the future.

So R isn't our destination. It's part of a journey that allows us to move towards what the best data science tool is in the future.

Why open source?

So just a bit of context setting for this, if you're not super familiar with Pharma, Pharma, in general, has typically used commercial software to support filings. I think that's been true across the industry, and it's been true for a really long time. That's kind of the way we've been doing things. And there's lots of historical reasons I won't get into there. But it does mean that our talent and tools, at least historically, were used for a lot of this. You know, we have a code base that's been built up over 20, 30 years, and a talent base that's been using that code base for this time. So that's kind of just to highlight why this shift is big for us.

So why are we doing this? There's loads of different reasons that have inspired us to make this change. I'm going to highlight a few today, but I wouldn't say that this is an exhaustive list. When it comes to R in particular, R is something that had been used at Roche for a while, but I would say the kind of wedge product that got it used a lot more at PD Data Sciences in particular was R Shiny. R Shiny really offered capabilities we didn't have already in the tools we had. The ability to make these interactive apps using code so quickly and simply really proved a really attractive offer and really encouraged us to start using R at least behind the scenes in clinical studies. And in fact, one of the earliest R packages that was built in-house to support this was a package called Teal, which Thomas is going to talk about later. This is now open source. You can follow that URL to go to see that package right now. And this was an in-house solution for standardized Shiny apps.

So we had this kind of justification for why we wanted to start using R more. And then there were a few other things that really started to weigh us towards thinking about moving towards R. One was just about the talent base. Generally speaking, when we're recruiting new graduates, they're much more likely to know these open source tools such as R or indeed Python than they are commercial software.

But the final point, which I think is a bit more future looking, is we wanted to go open source. And the reason for this is we think open source offers loads of opportunities for how we work. The first is just around getting latest developments more rapidly. Typically, if there's a new statistical method used to analyze data, that will often be implemented in R, first of all. So if we want to get access to those more quickly to provide better results, then we should be using R.

The other is we knew we wanted to go language agnostic. We knew we wanted that ability to switch between languages. And we know that open source is going to make that easier. Because with no proprietary formats, it's usually easier to switch between contexts. And the obvious example of this is the ability to switch between R and Python. There's been a lot of effort to make those two languages talk to each other nicely.

And the last one, which is one I'm really excited about, and Thomas is going to talk about a bit more in a minute, is the ability to collaborate with external partners. So up until a few years ago, all of our code was created entirely in-house. It wasn't really possible for us to work with people from outside of Roche. But if we go open source, we suddenly have this ability to get the inputs of people from across the industry, which I think is in the long run going to lead to much more efficient and better code and ultimately better outcomes for us and for our patients.

So hopefully that's been a good introduction as to why we're doing what we're doing. And now I'm going to pass over to Thomas, who's going to talk a bit about, as I say, the environment and the tools we're using in that environment.

Building the modern data science platform

Thank you so much, Kieran. So then let's talk about the building blocks of this kind of modern data science platform we set out to create, which is really focused on open source as its backbone.

So first of all, I would just like to quickly dive a bit more detail into the requirements we and PD data scientists have for our day to day work. And generally, we can separate our task into either the regulatory side of things or the exploratory side of things. And on the regulatory side of things, there's really four main deliverables we need to create. The first one is called STTM. So STTM is an industry standard for the data we collect in clinical trials. So regardless of how the data is collected in the end, when you submit it to the health authority, it has to be in this data standard such that it's harmonized across companies.

Next up, we have the analysis or atom data sets. And here you take the STTM data sets as source and then you transform them in a way that suits your analysis downstream. So as a quick example, imagine you collect a questionnaire which has 10 questions and an overall summary score. So the 10 questions you would have in your STTM data, because that's the raw data as collected. But then you would calculate the summary overall score as part of your atom transformation.

Next up, we need to perform a bunch of statistical analyses, and that can be something as simple and descriptive as counting adverse events or something much more complex, for example, fitting a survival model or fitting a mixed measures repeat model. And finally, we need to put those statistical analysis into an easily digestible format. And that's generally either tables, listings or graphs. And all these four components combined make up the packages that we submit to health authorities, such as the US FDA, in order to get market approval for novel therapeutics we're working on. So this is really at the core of our business.

But of course, we also have the exploratory side of things. And here, really, it's highly context dependent what we need to perform. And therefore, we need flexible tools and really the best tool for the job. So having an open source background where you have access to multiple language, for example, is really a good thing in this context. We also need to consider that there's novel data modalities we're getting more and more. For example, imaging data is becoming rapidly and more in our trials, digital biomarker, omics data, et cetera. So it's not just the classical rectangular data anymore, which we used to analyze in our commercial software.

And finally, we want not only to enable our data scientists to easily explore our data, but also people who don't work in data science, our colleagues in safety or clinical scientists. And we want to really enable them to easily explore their data and do so in a reproducible manner such that they don't always have to come back to us as data science. And we have this long feedback loop. And really, all of this combined necessitated that we created a next generation statistical computing platform where we could really perform all these analyses.

Introducing Ocean

And this is where Ocean comes into play, our one central analytics environment, which is the home for both our exploratory and regulatory work. So Ocean is built in the cloud on AWS, and it's a language agnostic platform, which right now supports our Python as well as SAS for our legacy work. But it is easily extendable because it's based on Docker containers. So if you spin up a container with, for example, Julia in it, suddenly you would have access to that language as well. For version control, we are using Git and GitLab. And we're not only using that for our packages we develop internally, but it actually will be there to store the code for all our studies. So if on a study you program a particular atom data set or an adverse event table, the code for that will be stored on GitLab.

Next up, and this is really a big change, even though it sounds somewhat silly, but we moved away from this proprietary format, SAS 7 BDAD, that used to be the only format really that we used to store our CDIS data. And instead, we opted for an open source format called Parquet, which is developed by the Apache Foundation. And it's a binary format for handling rectangular data, and it's highly efficient in both reading and writing, and it has served us already very well so far. We have an internal RStudio Connect server as part of this platform where we house our

exploratory Shiny apps, and we have a validated R package repository.

So while typically you would install packages from CRAN, internally, we use this validated package repository, and you would find a lot of packages that you know from CRAN, such as ggplot2 or dplyr over there as well, as well as our own open source packages we develop, plus internal packages. And it goes through our own internal, really streamlined and efficient validation process, which is called autovalidate.

Then we are using another open source tool called SnakeMake to orchestrate our production runs. And SnakeMake is very clever in a way because it creates a directed acyclical graph of all the artifacts that you need to create. So think of your SCTMs, your atoms, your TLGs. And then it really paralyzes as much as possible, taking dependencies into account. So what used to take maybe a whole day, for example, to run everything for a final clinical study report now is done in a matter of hours, if not minutes.

And finally, Ocean is the tool, is the home of our next-gen tools for clinical reporting, and I will spend the rest of my part talking about these next-gen tools.

OAK: automating SCTM mapping

So the first one of those is called OAK, and OAK is our solution for automating SCTM mapping. So SCTM mapping used to take a considerable amount of our time, and it was a rather labor intensive process that was also somewhat mundane. So we really opted to streamline this by creating an automation solution. And driven by metadata and our global data standards at Roche, OAK can now automate around 80% of SCTM domains with around 22 reasonable mapping algorithms. And this, in turn, then really saved us a lot of time. We've seen already efficiency gains by at least 50%, so this is really huge.

And finally, it's actually a fairly simple tool to use because it's, at the end of the day, a single R package you interact with plus a web app. But that being said, under the hood, it's somewhat more complex because it has to interact with several systems, and that's actually why it's called the OAK Garden.

And the first component of this garden is called Pistol and Honeybee. So really, what you have to think about is we have a metadata repository at Roche, which is a graph database. And inside that, we store the mappings from the raw data as we collect it to this SCTM standard in a machine-readable format. And that's really key here because that makes it then easy for R, for example, to query this information and make use of it.

Next up, we have our SCTM spec creator, which is called Mint. So this is a web application in React to create the study-specific SCTM mappings. And when I say that, really, most of it, at least 80% from what I had on a previous slide, should already be automated there if you adhere to standards. And then in this app, you have additional functionality to add any non-standard mappings. If, for example, you collect somewhat new data endpoints, which are not yet in our standards.

All of this is then saved in a repository, which is called SEPHRON. And again, in a machine-readable format, namely JSON, such that it may query down the road in R. And very importantly, any non-standards you would define in the Mint application, you would save in SEPHRON, such that you can reuse these non-standard mappings across different studies.

And finally, we have the actual data transformation engine, which is called OAP. And it's the R package. It's the R package people in the end would use. And really, it's a very streamlined process. You fetch all this metadata from SEPHRON, basically. And then you have a single function call per SCTM domain. And given all this metadata, it will then output for SCTM datasets.

So the Oak Garden is not open source. And the reason for that being is that it's fairly tailored towards our internal Roche architecture. But that being said, the overall principle is really broadly applicable to anyone who has to map SCTM datasets, which is also the reason why we're in active talks with the CDISC Open Source Alliance to create a proof of concept in which we would use CDASH rather than our own internal data collection standard as a source.

And if that's something you and your organization are interested in to collaborate, potentially, then please reach out. I certainly have heard a lot of questions regarding SCTM and R at RnPharma. So this might be a good opportunity to reach out. And maybe a good collaboration can spring out of that.

Admiral: open source ADaM dataset creation

So we cover the first pillar of our regulatory reporting site. And now it's time for the second, which is the Atom datasets. And here, our solution is called Atom, which is an open source modular toolkit for Atom dataset creation in R. And what is really unique about Atom is that unlike other packages we developed, which are now open source, this did not start out as a closed source internal project. And then later on got open source. Really, from the start of it, we collaborated with another organization, namely GSK, to hit the ground running and build something that would be usable not only for Rush's purposes, but really for the industry as a whole. And I think it's fair to say that this has been quite a success. We certainly want to use this kind of model of collaboration going forward.

And I think it's fair to say that this has been quite a success. We certainly want to use this kind of model of collaboration going forward.

And when I say success, I think one metric that really hits that home is that what started out as this collaboration between two organizations to create one R package, now is actually an ecosystem of R packages, which are developed by seven pharmaceutical organizations in total. So in addition to Atom, we now have Atom Onco, where Amgen and Bristol Myers Whip joined us with their oncology expertise. There is the Atom Opta package, which Nobartis and Rush are working on together. And finally, GSK is collaborating with Pfizer and Johnson & Johnson on this Atom vaccines package.

And if you and your organization are interested in leveraging R and Atom and Admiral for Atom dataset transformation and you have expertise in a particular therapeutic area, again, please reach out. We would really love to see this ecosystem growing.

Statistical engineering and TLG creation

Next up, I want to talk about our statistical engineering team, which is really working at this intersection of biostatistics and statistical or software engineering, actually. So this team really focuses on accelerating the adoption of novel statistical methods. So really all the cutting edge stuff that comes out of academia and trying to implement that as software. And so far, this has been mainly focused on R packages and also occasionally standard or C++ have been used in the background. But the team is really open also to other languages, for example, Julia, basically anything that is open source and a viable solution for statistical computing.

And really important to highlight is this this team is not only working at Roche alone, but it's part of a larger industry wide effort, namely the software engineering working group of the American Statistical Association, where not only Roche and other pharmaceutical companies, but also academic institutions jointly work together to basically implement the cutting edge of statistics in software. And here's just a couple of examples of packages the statistical engineering team has worked on. The MRMM package down there is actually the first one that came out of the software engineering working group. And another one I would like to highlight is this RBMI package for reference based model imputation, which won an award by the Roche Statistical Society last year. Once again, this has been a collaborative effort in this case between Roche and the University of Bath, so an academic institution.

So finally, on the regulatory side of things, we have the creation of a new regulatory side of things. We have the creation of our tables, listings and graphs. And really, when we started doing that in R, the G part, the graph part was the easy one. We just used gtplot2. It was the best tool available for the job. It still is, in our opinion. And it was certainly also a big reason why to shift to R in the first place, having such a good tool readily available. However, the T side of things, the tables, especially if you look a couple of years back, there was not such a really good solution out there. So what we did is we created our own, which is called RTables.

And RTables really has an expressive and modern API for table generation in R. And very importantly, it is built from the ground up with the needs of clinical reporting in mind. And we think that this is really key because the kinds of tables you do in this space tend to be much more hierarchical and complex than, say, a typical table you would find in an academic journal or even on some websites. Right now, RTables supports outputting tables in plain text, Word, PowerPoint, PDF, HTML, and the version on GitHub, the development version right now, also added RTF support. And this is an open source package which you can install from either GitHub or CRAN.

Something that is working in tandem with RTables is TURN. And it's really a complement that focuses around the summary statistics because you can think of it as bridging the statistical analysis you perform, for example, with one of the packages we've seen on the previous slide, and then actually getting them to display in the table. And this is yet again an open source package which is readily available for you also to use and check out.

The final piece on that kind of TLG side of things is Chevron. And this is really about streamlining our processes even further. So when we started out doing tables in R, we had an internal TLG catalog, which looked something like this. This is a particular first event table. And you can see it's just kind of a code dump you would copy-paste into your study folder and then use. But really, copy-pasting is not a good solution. So we opted for something sort of template-based. And right now, you can basically create the same table by using the Chevron run function, indicating the template you want to use, and then feeding in your data. And off you go. So a further gain in efficiency in that way.

Teal: exploratory Shiny apps for non-developers

So finally, let's talk also about the exploratory side of things. So I mentioned in the beginning, we want to enable our non-data scientist colleagues to also easily and, importantly, reproducibly analyze their clinical trial data. So we set out on this journey and said that R Shiny would be really a good solution for that. However, we just have a problem, which is, how do we actually get our 600 or so data scientists who are not Shiny developers to build these exploratory apps in the first place? We did not want to train all of these 600 people to become sort of Shiny gurus. Instead, what we opted for is to create this framework called Teal, which Kiran already mentioned in the beginning. And this framework really centers around reusable modules. And it really abstracts away a lot of the complexities of building Shiny apps from scratch.

You don't actually have to care so much about the UI as well as the server logic in the modules you would readily adopt from the catalog. This is an example of a very simple app created using Teal. And you can see on the top, you have all these tabs and each of those tabs represents a particular module. So in this case, it's a demographic table module. And what all nodes have in common is that in the center, you have a display. In this case, it's a table, but it could also be a plot. And then on the left, in addition, you have this encoding panel where you can kind of customize what you see in this table. For example, here, you could select different variables to be displayed. So that's all the module specific part.

And as I said, we have a large catalog, I think even over 100 modules readily available for our internal people and also for you, because this is open source to use. But then the Teal framework itself also provides a couple of goodies that work across all modules, the first one of which is this filter panel you see here on the right. So depending on which data is used in a particular module, it would list out the data set on top. And then you have the ability to select any variable out of these data sets to drill down the data. So in this case, we have a particular subgroup based on ADSL. But if you imagine this would be some kind of adverse event analysis, you could easily filter down for, let's say, serious adverse events.

Next up, we have this reporter functionality, and this is a really neat one. Anything that you create in any of these modules, you can add to a shopping basket, sort of. And then at the end of the day, you can go to the card and check out by exporting a PDF report or a PowerPoint report, which would then contain everything that you added to the card. So maybe a demographic table, a couple of my upload, a response table, whatever you may wish.

And finally, we have this show R code button functionality. And this is really key for us because that enables the reproducible nature of Teal, because unlike Tableau, for example, or Spotify, which are also great tools to visualize data and drill down. This is really 100% reproducible because if you click the button, you will get a complete R script, including loading all packages, loading all data that is needed, filtering the data as you have in the filter panel here, and then reproducing the exact display that you see here. And that's really, we think, a key for us.

This is really 100% reproducible because if you click the button, you will get a complete R script, including loading all packages, loading all data that is needed, filtering the data as you have in the filter panel here, and then reproducing the exact display that you see here.

So I already mentioned this has been open source since last year, and the code is available on GitHub for you to look at and also to install it from there. We have two sample apps which are publicly available, and the previous picture I showed is one of those. So definitely check those out. And actually, Teal has been used in an R submission pilot to the FDA, which is quite an exciting development, kind of instead of the static TLG submitting a shiny app. And there will be more communication about that in the future.

We have an upcoming webinar around Teal at the end of February, where we will dive deep into this particular tool for an hour or so. And finally, we are looking for collaborators to develop this framework further. And I think that really goes to all the tools that we've shown here, kind of in the spirit of the open source community, where we take a lot of the great stuff that other peoples have put out there. We also want to put the packages we develop out there and collaborate with others to develop them further.

And finally, I would just like to mention that this everything sort of I said is part of this larger effort in the industry called the pharmaverse, where different pharmaceutical companies come together and develop R packages for end-to-end clinical reporting. That's really a great development, which we really hope further gains traction this year and the years to come. And with that being said, Kieran, I hand it back to you.

Oh, thanks, Thomas.

So yeah, Thomas has spent some time talking about the sort of setup of how we're going to do our rush. So we've kind of it's not all finished, but it's very close to finish. You know, it's a work in progress where we have this environment we're going to work and we have this tool base. And certainly both of those are going to continuously evolve over time. But we have a good idea of what we're doing there. And we've presented that now. But the other piece that we really can't forget about when it comes to moving to a new language is looking at our talent base, making sure that they're ready for this change.

And it definitely is a big change. You know, it's not just about shifting to R. We're changing the environment we're working in. We're changing the tools. We're using version control. So there's a lot of things for our talent to learn. And we need to make sure that they're supported.

So what I'm going to talk about for the next 10 minutes or so is just about some of the measures we're taking to support everyone. And this is all, you know, this is a work in progress. This is something we're learning with as we're going. And certainly I expect to be giving future talks in a year's time where we talk about what worked and what doesn't. But here's at least some of our ideas that we're kind of throwing out there right now.

So just thinking for a second about who we have working in PD Data Sciences. There's a real mix of different people here. We have people who've been in the industry for 20 or 30 years and have never coded an R at all. So they have a wealth of experience in clinical trials, but they don't have much experience in R in particular or Git as well. And then we have people who are our experts. We've got a lot of people who've joined the company more recently who, as I kind of alluded to earlier, have this R experience and haven't had the opportunity to apply it in the context of clinical reporting.

The other challenge we have is that not everyone's transitioning to R at the same time. So again, this is something I mentioned in the intro that we're not, you know, we're not moving every single person to R all at once. We're doing it in a phased way. And that's good from a management point of view, but it does mean that you can't just give everyone training one day and expect them to be able to apply it the next day.

And the other thing that I think is really important when you think about transitioning to a new language, it's not about just learning kind of how to use that language. It's also about being fluent in that language. We really want our programmers to start being able to know what good R code looks like, not just be able to throw something together that just about works.

We really want our programmers to start being able to know what good R code looks like, not just be able to throw something together that just about works.

Guiding users through self-study

So as I say, I'm going to go through some of the solutions we've looked at for dealing with these issues.

So the big one, I think, is around helping guide our users. So because of all those challenges we have, doing active live training isn't always going to be possible. Realistically, in terms of resource restraints and the fact that everyone's not transitioning at once, it doesn't make sense to do live training. It's certainly something we'll do, but we need people to be able to do self-study. And the problem with self-study is that if you just put someone in a room and tell them to learn a subject, they might be able to do it, but it's going to be challenging. They're not going to know where to start necessarily. And in particular, they might go on a journey that helps them a bit, but isn't as guided as it should be.

So we've been looking at a few different ways to make this more effective. The first idea is to work out what people don't know. Often when we've been sending out initial surveys on people's knowledge on subjects, we do it in a kind of granular way. We say, you know, how long have you been programming this language? How good do you think you are on it? And self-assessment on that level can be helpful, but it's often true, particularly when you're starting a subject, that you kind of, you don't know what you don't know. So there may be gaps in your knowledge and you're just not aware of them because you're not familiar enough with the language. And certainly for those of you who are familiar with R, there's lots of pitfalls when it comes to the complexity of how R works, that if you're only starting off with R, you're not going to necessarily know about.

So that's what the data science profile does. It breaks all of these categories into discrete topics and then builds assertions on kind of knowledge we expect people to know to be considered proficient in that topic. So I've got a screenshot, which is an example here for Git. And it's got a collection of different things I might expect as basic knowledge when you're starting in Git. And basically, this is all based on self-assessment. So we're not trying to catch people out here. We're not trying to trap anyone. We want an honest reflection of where they are. And that helps them to understand where they need to focus their learning on.

The Data Science University and curated learning paths

So the next step is giving them the resources to fill those gaps that they have. One important thing we've kind of thought about is we don't really want to reinvent the wheel. What I mean by that is there can be a temptation when you've got some time and the ability to create training materials to go ahead and just start making training resources without thinking maybe this resource exists already. There's a huge amount of material online, both free and available via commercial license. And some of it is great.

The challenge that a new learner has is they don't necessarily know how to direct themselves in this. So we've already got a bit of a start, and they know the kind of topics they want to work on. But even then, what are the best resources for them to use? And what orders should they be progressing through them in? So what we've built for this is an in-house solution, which we're calling the Data Science University. And what this is, is it's a tool to curate existing materials. So what we do, and we've got an example here, is we build learning paths on a particular topic. So this is a good coding in R topic, where we pick out, we pick that into discrete topics. So again, that links to that data science profile of seeing where your gaps are. And in each one, we provide material that we think is of high quality for learning about that subject. And again, that may be an in-house training that we've created, but it could well be a great resource that's available for free online.

So the other thing I wanted to mention as well is, obviously, a couple of these things are in-house, and you may be curious about some of the material we're producing at Roche. Well, I wanted to take a little opportunity to do a sneaky advert for something that's coming out very soon on Coursera. We've actually been building a Genentech-based course called Making Data Science Work for Clinical Reporting. This is actually going to be part of a wider specialization, and it's going to look at how we think data science should be done in the context of clinical reporting. So this course is literally being beta tested this week. So I'm really hopeful that it's going to be out either this month or early in February. So look out for that if you're interested in the kind of material we're producing around learning how to use data science in the context of clinical reporting.

Making training relevant and timely

So when it comes to making training resources, one thing that I think is really key that's kind of been learned by hard experience is try to make sure it's relevant. One experience we've had when we've, for instance, paid for external trainers to come here is that training material might be really great, but it doesn't take advantage of the fact that we kind of know what we do for a living, right? We know what our programmers are doing in their day-to-day work. So when we develop trainings, we can make it relevant to what they're actually doing. It definitely involves an effort, but I think you get a lot of value by making problems real. One simple way of doing that is just picking out real data, not using empty colors or iris, which is what you'll find on online resources. And that's what we've done in the trainings we've built in-house. There's actually an open source package that we've created specifically for generating synthetic CDISC data, so you can make use of that right now, and that's the link on the screen there.

So the last point on training is making sure it's at the point of need. So this is kind of relating to what I was saying earlier on, that not everyone is moving towards using R all at once. So we're doing our best to focus our training on those who need it. We're training in cohorts rather than trying to train everyone at once, because if you take a training a year before you actually apply it, you've basically wasted your time because you've forgotten everything you're using in that time period. So our goal is training at the point of need.

Building a culture of continuous learning

So the last point I want to make, which is going to lead into Ning talking about the business transition, is that while I've talked a lot about training here, learning about is that while I've talked a lot about training here, learning really isn't about training. So that relates to that last point. Realistically, how you learn is by applying all of those trainings in your actual day-to-day work. So we want to encourage that kind of context learning.

And as well as that, we also want to build this culture where you are always looking to improve how you code. So this is speaking back to this idea of not just making people learn a language, but also becoming fluent in that. And there's a few different things we're focusing on to try and support that. One is about building good frameworks to apply knowledge. So in this environment we're building up, making sure we have good frameworks where we encourage people to follow best practice just by those frameworks.

The other is around learning via code review and seeing other people's code. Certainly, one of the ways I became a much better R programmer was by reading code written by someone who was a better programmer than me. And I had code that worked, but when I reviewed their code, I realized that I could be doing things much more efficiently and writing code that's much clearer to read and easier to use in the future. So we're really going to encourage this culture of code review, of looking at code and seeing how we can do better.

And one connected idea with that, which was actually inspired by the tidyverse days run by Posit, was having these internal hack events, which we called HackR, where we basically took all of those tools we're building that Thomas went through and highlighted issues that we thought people could take on. So we said to our wider data science audience, get involved. You can improve these tools. And this is another way of highlighting this idea of an open source culture, the idea that these tools that we're using and developing to do our work, they're not made by wizards in the sky. They're something we can work on and improve, and we can collaborate together to lead to better code for all of us.

These tools that we're using and developing to do our work, they're not made by wizards in the sky. They're something we can work on and improve, and we can collaborate together to lead to better code for all of us.

So the last thing which is connected to that is learning as teams. So I mentioned that we're not transitioning at once. So when we're bringing teams as a whole into this new environment, we're going to make sure that they learn together, because I think another way of learning quickly is by learning with peers. So we're going to make sure that teams upskill together. We're going to support people on their journey with additional mentoring, where we share knowledge from individual to individual as well. So I'm going to pass now to Ning, who's going to talk a bit about how that transition for those teams is going to work.

Business transition at scale

Thank you very much, Kieran. Yeah, so I'll take over and talk about how we are making our business transition. Yeah, so as Thomas already alluded a little bit, like from Roche, we're a huge company. Within Roche Data Sciences, we have over 1,000 employees, and we are working on over 200 active modules. So as you can imagine, we have a really diverse group over there with different skill set, different background, and also we have very diverse type of projects.

And in addition to that, also, as you may already get a feeling from Thomas' presentation, that for this transition, the scale is very large, not only because we have a large team, but also because there are so many new tools available out there, including the R packages, as well as some non-R tools, including Git, Docker, Cloud Computing, and Snakemake. This is the beauty of open source, but also you can imagine that that introduced additional complication to our business transition.

So to ensure a successful business transition, we realized that we really need to make sure that we can empower every single project team to make their own decision and also to make their own fee-for-purpose transition plan. We're kind of thinking that for each project team, when they try to modernize their data science tools, it's sort of like renovating their house. So as we all know that when families decide to renovate their house, each family has their own needs. Some families may spend more time on renovating kitchen and bathroom, and maybe other families will spend more time on renovating living rooms. And for people who are purchasing a new house, they probably will pick out the new tools and new materials when remodeling their house.

Similarly, for analytical projects, we're thinking that during this transition journey, for some of the analytical project teams, maybe they think that it's valuable to refactor everything, including data process, data analysis, etc. And maybe for other teams, their data processing pipeline is very stable using the existing tools, and they prefer to keep using the existing tools for data processing, but only adopt the new tools for data analysis. And for brand new projects, we think probably likely the project team would love to start from the newer tools. So we want to enable each project team to make a plan that are tailored to their need.

In addition to that, similar to remodeling, each family probably have different best timing for their remodeling. So for analytical projects, we also think that different teams should have the flexibility to pick up a good timing for their team to adopt this transition. Likely that that will be the time frame when they don't have too many business critical deliverables. And lastly, of course, different analytical teams have different access to talent pool and resources.

Although we want to empower each individual team to make their own decision, as a big organization, you can imagine that like it's probably you will probably create a lot of chaos if we just go think push things in a decentralized way and ask everybody to make their own decisions. So as an organization, we still need to want to have some light coordinations across different

So at the end of the day, we really want to come up with the adoption role map for the next couple of months.

With that, we will have a holistic understanding about how many teams are planning to be onboarded in a certain time frame and what are their needs, what are the tools they need to learn, etc.

In addition to that, by having such an adoption role map, we also think this is a very good opportunity for us to build some mechanism to enable iterative feedback to our two developer team. We're hoping that for each wave of onboarding, there will be a mechanism to feedback to the two development team so that we can keep improving our tools as the onboarding goes.

So how can we achieve that? How can we come up with such an adoption plan? So from Roche, we have a small group of people coming together, created this adoption scheduling team, and we work together with the project teams trying to work out this adoption plan. So the project teams will still be the final decision maker, but we have this scheduling team over there trying to coordinate the onboarding timing across different teams and also trying to provide consultations for them to understand the tool availability, etc.

Building the adoption roadmap

So what are some of the considerations when we build up our adoption role map? The first thing is definitely business continuity. We want to make sure that for each project team, they can identify a couple of good transition timings for themselves and we can stick with those good timings so that we can minimize business interruption when doing this transition.

Secondly, we also want to take this opportunity to enable early collaboration between the developers and also the users and to allow iterative development. Therefore, we decided to start our adoption at a very early stage, even before all the tools are 100% ready. So we engaged a small number of users when the tools are still somewhat half-baked, and we ask for feedback and we also encourage the users to actually co-create with us by filing bugs or by even contributing code back to the code base.

The other thing we noticed from past experience is that if we look for early tester volunteers, sometimes we are getting those expert users being very active to be the early testers, but those users may not represent the majority of the user base. Therefore, when planning out this adoption role map, we reach out proactively to different teams and try to include a diverse type of project and user base in each wave to represent our user base.

In addition to that, we also want to make sure that when we work out this role map, we want to ensure that the trainer and support capacity are considered. We don't want to overload our trainer or the support team in a short time frame, and we want to make sure all the users get sufficient support.

And lastly, in the discussion between the scheduling team and also the project team, the scheduling team also provides consultations to help the project teams identify relevant tools and processes to be adopted. And as mentioned, not every single study needs to adopt all the tools out there. And also, as mentioned, for early adopters, not all the tools are ready when teams start to onboarding, so our scheduling team also makes sure that the project team's adoption timeline aligns with our tool availability.

Community driving the transition

So to support this journey, as you can imagine, this is not something that can be driven by a small group of people. Although it is Thomas and Kieran and myself presenting here today, there is really a large community behind who is driving this transition within Roche and also in the pharma industry. And we really want to say thank you to everybody who got involved.

And I also want to mention that for many people who are driving those activities, they are not spending 100% of their time on developing a software or spending 100% of their time on the change management part. Rather, the majority of people are still working on molecule teams, and they're spending 20% to 30% of their time on driving certain aspects of our transition here.

So what are some of the communities that are driving this transition within Roche? The first one is definitely the software product teams. Those are the developers who wrote the R packages, who built the infrastructures, who wrote the business tools. We also have a ways of working community, and those are the people who are looking at how we can integrate the new tools with existing business processes. And they also come up with new guidelines in terms of ways of working using newer tools.

And there is this roadmap scheduling community, a group of people who interview with the project teams and provide consultation to the project teams to come up with an adoption roadmap. And when it comes to come closer to a project team's onboarding timeline, they will be assigned with an onboarding coach, and the onboarding coach will help them to identify certain training materials that are relevant to their team, also point them to activities such as hackathons Kiran mentioned. And there is a user support community answering the questions from the users. And lastly, there is a communication community who share our adoption progress to the wider community and also share with people what to expect as the next step.

Current progress and rollout timeline

So where are we right now? During Q3 to Q4 last year, we welcomed a couple of teams joining us as early adopters. We had two waves of early adopters join us. And as mentioned, for those early adopters, when they join us, our systems, our tools are not 100% ready yet. And they're really the trailblazers helping us to test out the tools and have early interventions, giving feedbacks, et cetera. And for those project teams, they did have some backup plans by using the existing tools because the newer tools and systems are not 100% ready yet.

So in Q2 and Q3 this year, we will welcome a couple of new molecule teams joining us. So for those new molecule teams, they are the teams who don't have any legacy code. So they will start everything from scratch. So as you can imagine, they will be using the new tools end-to-end in an end-to-end fashion, and they will build this whole end-to-end process using our newer tools. And we're really excited to see how that testing goes.

And after that, if everything goes smoothly, in Q4, we expect to welcome more studies, more project teams joining us for a wider rollout. In that rollout, we will welcome some of the new studies, and also we'll welcome some other projects who already have some existing code, and they will pick the part. They might decide to refactor some of their code, or maybe they just decide to migrate to the new system without refactoring their code.

So the journey continues, and we're really, really excited to be part of this journey. And hopefully, if we got a chance to speak again in a year or so, we can share more good news. So with that, I think we are happy to take any questions.

Q&A

All right. Awesome. Thank you so much, Rosh and Genentech team. Lots of wonderful information. Tons of questions are coming in. I know we've got about six or seven minutes here, so I'm just going to jump right in with a couple of questions that are floating to the top.

So the first one I'm seeing is, what hurdles did you have to overcome for transitioning from SAS 7 BDAT to Parquet files for data storage?

So yeah, so this is an interesting one, because Ning just mentioned we'll come back in a year and we'll tell you success stories. So some of this is work in progress. These are things we're trying out right now, and we haven't got too much experience of doing it. So this is our intention, and we have some early teams that are trying out storing in Parquet. And so far, that's working really well.

There's obviously going to be a tension when you're storing things in Parquet or in SAS 7 BDAT, because you've got to get SAS to be able to read that files. But we're building solutions for that to be able to happen. But I mean, at least for natively using them in R, we're quite optimistic about it, because Parquet is a really efficient format. And just in our testing, it seems to be that it's going to be a really good format for not only for R, but also if we use Python, Python can read Parquet natively as well.

Fantastic. Any other comments on that one?

Maybe that's not related to this particular question, but somehow there was a question around whether in the end we still have to submit these XPT files, which is a SAS format to the FDA. And actually, indeed, that's the case. So this is something we use internally for data storage. And once the data makes it to the FDA, we'll use XPT, or maybe if the FDA approves this new format, which is being proposed, data set JSON.

Fantastic. All right. Let's move on to question two. Can you speak more about the capabilities Shiny offers that you did not have?

Oh, very open-ended. I think really one key aspect for us was around this reproducibility I mentioned. Because, for example, Tableau or Spotfire are stuff that we use internally. It's certainly very powerful, and you can do lots of stuff. But if a safety scientist has such a dashboard and comes with you, this is something I want to have in my CSR. You have basically no idea how they reach to that conclusion. Which subgroup is this that you see on this plot? How did they fill the networks, events, et cetera? So that really was, for us, a big push.

And also, we wanted to have something which bridges this gap between the regulatory side of things, or classical tables, and the exploratory nature. So a demographic table I showed you basically looks the same in the app as it would in the final CSR. And thereby, it's really a nice match between those two.

And also, we wanted to have something which bridges this gap between the regulatory side of things, or classical tables, and the exploratory nature. So a demographic table I showed you basically looks the same in the app as it would in the final CSR. And thereby, it's really a nice match between those two.

Got it. Awesome. Any other comments on that one?

Oh, and maybe just to add, so for example, I'm not sure how much statistical analysis functionality Tableau or Spotfire have. They're mostly for visualizing data. But for example, in Teal, you can fit an MRMM model, which is something fairly complex. And I'm not sure, besides SAS and R, which other languages even can do that. And certainly, I don't think any dashboard app out there has the capability.

Right, right. Awesome. Let's move to question three. Is there a balance of using Python and R in your strategy, or a wholesale decision for R? If the latter, how is that made?

So I think the way to look about it is the backbone of our data science is going to be R. So our focus of developing tools is going to be R. So when we look at our resource of our actual teams, building the tools that we're going to use, that's going to be in R. And that's kind of the natural outstretch of that is that most teams are going to be using R because that's what the tools have been building. But there will be the ability to use Python where the problem is more easily solved in Python.

As to why R, I don't know if I want to say anything too controversial about the difference between R and Python. But I think it's fair to say that as a statistical piece of software in the open source, I would argue that it is at least equivalent with Python when it comes to statistical problems. And there's a lot of benefits in terms of just like what our talent base knew already. Yeah, I don't know if anyone else wanted to say anything else about it.

Yeah, I totally agree. I feel like right now, probably in academia for advanced statistical methodologies, like many of them are still developing in R. And yeah, for example, MMRM, et cetera. Yeah. So I think like right now, there's more tools available in R for some of the statistical methods.

Got it. All right. Going to move to a popular question that comes up quite a bit. What kind of validation is required for using R, Python for clinical trials reporting?

So yeah, I think Thomas mentioned already the auto-validate tool. But I should say that that comes from a history of collaboration across the industry. So there was still going the R validation hub group, which I think is still led by Andy Nichols. It was initially anyway, which looked at these kind of questions as to what is R validated? How do you validate an R package and those kind of questions from it?

So it was kind of a collaboration, taking the sort of outcomes from that and building those into our internal solutions, working with our internal validation teams to get consensus. And yeah, the auto-validate R was a real late forward for us because it was an in-house solution for validating R packages really quickly, effectively, automatically. That's why it's called autovalidate, in that we have these standard measures of an R package quality that we can use to assess whether we think an R package can be included. And I mean, if it doesn't meet those standards, then we have to do some action. But if we get an R package and it meets all those standards, then it can be included in our suite of packages kind of straight away, which was a real leap forward for us.

All right, we're almost at the hour here. So we're going to go a little fast on these next two, but let's squeeze them in if we can. The next one is how much manual work do you have with this approach between raw data and

STDM and Atom? So the piece from raw data to STDM, as I mentioned, we are getting around 80% or so automated, which means you only have to take care of the manual 20%. You would go into this Mint app and then basically say, OK, this is how I want to map this raw data into this standard. And if you have done it for one study, you can actually then reuse it again for the next step, thereby again gaining efficiency. So this should be really fairly streamlined.

Then the next step from STDM to Atom, that's much more difficult because it's not just a simple mapping. Depending on the analysis you want to perform downstream, you need to derive certain things. So while we have template code available that implements sort of our general standards, there will always have to be extensions. And this will be a much larger part of the work compared to STDM.

All right, anything else on that one?

Version control and data storage

Two more questions we'll knock out here pretty quick. All right, question six here. Do you keep the datasets under version control in GitLab, et cetera, along with the code? If so, do you have issues with large data files? We can just say no, no, we don't. So we have a separate solution, which I think Thomas mentioned, using S3 for storing our data. Yeah, I think it's fair to say if we did store on GitLab, yes, we would have a lot of trouble.

Roche ecosystem and the pharmaverse

Will the Roche ecosystem of apps work with the pharmaverse suite, or is it one ecosystem versus the other? It's an interesting way to put it. So more or less everything I talked about is actually part of the pharmaverse. And the pharmaverse is, you don't have to think about it as kind of us putting it out there. There's lots of companies contributing various stuff. There's actually our tables, for example, is part of it, but there's other table packages as well. So it's just kind of a curated list of packages that adhere to a certain set of standards and that are useful for clinical reporting. And most of them work independently of each other, so you can mix and match whatever you want to use. And I don't really think there is like a Roche kind of app suite, kind of just the pharmaverse out there, and we're using as much as possible out of that.

So it's just kind of a curated list of packages that adhere to a certain set of standards and that are useful for clinical reporting. And most of them work independently of each other, so you can mix and match whatever you want to use.

Auto-validate for packages

I think we got one more question here that was quite popular. There was one that came in about Parquet. I think we got to that one, so I'm going to skip it. But the next one is, can you please elaborate on the auto-validate for packages? I think there was some interest on that.

OK, so I think, yeah, so the basic idea is it's a tool that can generate reports, the kind of standard reports we need for validation. So those are generated by the tool. And what you do as a user, as a creator of an R package, is you submit your R package and you make sure it meets a sufficient quality check. So we have a list of different quality checks that it does. I don't know off the top of my head what they all are, but they include things like obviously passing CRAN checks, having test coverage past a certain level, you know, all these kind of sensible things that we think denote an R package being of sufficient quality. And if you pass those, then auto-validate does the rest of the work for you.

And there was a great talk by our colleagues at R in Pharma just on this topic, I think like 20 minutes. So if that's of interest to you, definitely check that out. Yeah, and you can get that talk and others on the R in Pharma YouTube channel.

I think with that, I'm going to thank all of you for such wonderful information. I know there's a lot of excitement about the webinar, Thomas, that you mentioned that's coming up. I think they can get that. Is that through the R Consortium webinar series? Yes, it is. And it's not yet on the website, unfortunately. But February 24th is the date. Teaser.

All right. Yep. Keep an eye out for that. And with that, I will wrap it up. Thanks for all the great questions and comments. It's great to see the community come out. Thanks again for all the effort and time to put this together, especially right after the holidays. And with that, I'll wish everyone farewell and have a great day, morning, evening, where you are. Thanks a lot.