image: thumbnail.jpg
Benedikt Kahmen @ Generali | Data Science Hangout
Speaker bio: Benedikt is the Head of Analytics, Data & AI at Generali Deutschland AG, one of the largest primary insurers in Germany. He has spent the last 10 years building and scaling data science teams in insurance. Together with his teams, he launches and operates a wide range of data science applications from marketing and sales to customer service and claims in a heavily regulated environment. Before coming to insurance, he completed his doctorate in philosophy at the universities of Bielefeld and Oxford. He holds an M.phil. in philosophy from LMU University of Munich.
________________________
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Hello, everybody. Welcome back to the Data Science Hangout. If we haven't had a chance to meet yet, I'm Rachel. I lead Customer Marketing at Posit. I'm so excited to have you all here joining us. If it is your first time, the Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. We get together here every Thursday at the same time, same place. I think we've been doing this for about two years. Is it over two years now? But again, if it is your first time joining us, so nice to meet you. Say hi in the chat if you want to welcome. We'd love to welcome you in to anybody joining the first time.
But we're all dedicated to keeping this a friendly and welcoming space for everybody and love to hear from you no matter your years of experience, titles, industry, or languages that you work in. It's also totally okay for you to just listen in here if you want. Also awesome to be a part of the party that you'll notice happens in the Zoom chat here. I love to see people meeting each other through the Zoom chat and sharing resources for each other with each other there. But there's also three ways that you could jump in and ask questions or provide your own perspective today. So first, you can raise your hand on Zoom and I'll call on you to jump in.
Secondly, you could put questions into the Zoom chat and just put a little star next to it. If it's something you wanted me to read out loud, maybe you're in a coffee shop or something. And then third, we also have a Slido link where you can ask questions anonymously. And quick note for anybody watching this recording in the future on YouTube and if you want to join us live, the link to add the event to your calendar will be in the details below. If you are adding the recurring event, just double check the time zone for you. I know there's been a few issues with certain time zones and just making sure it gets added to your calendar correctly. So it's every Thursday from 12 to 1 eastern time. No rule that anybody has to stay on the whole time or talk. Just come and go as it fits your own schedule. But with all that out of the way, welcome again. So excited to be joined by my co-host today, Benedict Common, head of analytics, data, and AI at Generali.
And Benedict, I'd love to kick things off by having you introduce a little bit about yourself and your role, but also something you like to do outside of work. Yes, sure. Thank you. Yeah, I'm Benedict. I'm the head of analytics, data, and AI at Generali Deutschland. So that's the German branch of the Generali Group, which is one of the leading primary insurers in Germany. We're working with a bit over 9,000 colleagues here. I've been with the company for the last 10 years and had the amazing opportunity to build a data science department from scratch, from literally one or two colleagues to now three full teams. Before that, maybe somewhat unusually, I did a PhD in philosophy. So if you're interested about that, just go on and ask me.
Yeah, and something outside work, I also am a marathon runner. I think last week we also had a marathon as a topic. So yeah, I'm doing that as well. Awesome. What's your next marathon?
Hopefully Luxembourg. I live in a part of Germany that's close to Luxembourg and they do a night marathon there. So maybe that's a good challenge. That's awesome. I'm curious because you mentioned your background in philosophy as well. How has that influenced your approach to building and scaling data science teams? We start with the easy questions, right?
Well, I just, it kind of tipped me off right when you said it. But yeah, just curious like what the intersection is between the two as well. Yeah. So maybe the intersection first. I did my PhD in a part of philosophy that's called philosophy of minds, which tries to understand how thinking works. And well, one way to do that is to think a lot, which is what philosophers do. Another way is to do experiments, what psychologists do. And a third way, the engineering approaches try to build a mind, which is what data science and AI researchers do. So it's not really that far removed. It's the part of philosophy that's maybe closest to data science and AI. Although I've never seen one bit of real data in all of my philosophical studies. So that's part of the overlap and something that comes up every now and then.
How has it influenced my approach to building teams? Well, maybe not the philosophy background, but the academic background has influenced me there because my desire to study that and do a PhD there was always driven by curiosity. And I think that's one of the central values that's like the quality of life for data scientists in the day work. So I'll try to give as much room as possible for this exploration, this curiosity also in my teams. Yeah.
my desire to study that and do a PhD there was always driven by curiosity. And I think that's one of the central values that's like the quality of life for data scientists in the day work. So I'll try to give as much room as possible for this exploration, this curiosity also in my teams. Yeah.
Data science across the organization
Love it. Sorry, I started us off right with an in-depth question there, but maybe I'll take a step back here. But I know you mentioned that you built the team up from one or two people. Can you just share a little bit about what does data science look like across the organization now? And what is the team that you lead? How many data scientists are there?
So we have major data science teams in four or five functions within the organization. Being an insurance company, of course, we have data scientists in the actuarial departments, which, for example, are responsible for the insurance prices. Then we have also data science teams who work across different functions. And my teams are part of that. So we are part of the chief customer officer function, which is like the umbrella for everything that has to do with distribution, but also with the customer contact and the customer lifetime, optimize everything that we get our hands on.
The size of the team, so it's currently 23 people. And with that, we are, I think, one of the largest, maybe the largest dedicated data science team in the general group.
Thank you. Just to all put ourselves in your mindset and thinking about the work your team does, would you be able to maybe share an example use case of something? Or walk us through that?
We have a number of use cases across different functions that we touch. So for financial services, firm, of course, marketing distribution are very important. So we spend a lot of time doing a marketing mix modeling or attribution modeling just to find out how to optimize our marketing spendings and distribution activities. Very closely linked to that are use cases within the customer relationship management area. So for example, calculating next best offers, the next best actions for customer agents is one of the things that we do. At the intersection to the actual area departments, we also develop models that inform the pricing that we do. So we are involved with price elasticity models or with survival and retention modeling. And we also have quite a big interest in operations, which is for a services firm again, a lot to do with extracting information from unstructured data, a lot of paperwork actually, but also knowledge retrieval. Generally has been in the business for quite a while. So we have very old insurance conditions sometimes, and you get very complicated questions that you have to answer about what is insured in a policy that's 20 years old and have corner case there and find that information really quickly and use that to optimize resource allocation.
So routing customer intents to the right person who has this knowledge and helping that person find the knowledge even faster. So that's quite a broad range. If anything is of special interest, just let me know. Yeah, I always love hearing about the customer journey or customer experience examples. I I've just always been a little bit obsessed with that customer experience, and I've always wondered, like, even for things like the data science Hangout, like, how does somebody attending the Hangout impact their experience as a customer to.
R and Python in the team
Okay, let me remind everybody who just joined just how you can ask questions to if you want to jump in here so you can put questions right into the zoom chat here and I'll call on you or if it's something you want me to read instead, you can put a star next to it and then we'll reshare the Slido link in just a second here where you could ask questions anonymously as well. But Benedict, hearing a bit about the team, I was just wondering, do you feel that the team has like an even split on R and Python or more people starting to move towards Python? I know previously, like insurance has seemed heavier in R, so I was just curious. Yeah, so right now I think it's an even split or roughly an even split with an unclear tendency or with no tendency. So it stays about the same. We have the requirement for every data scientist who joins the team and only speaks for one of the two languages that he or she learns the other within the first, roughly first year, so that everybody is comfortable with working in both languages.
Initially, we have tried out projects where within one project, we also mix the languages, so write one module in R and another in Python and a third sometimes in Java. We don't do that anymore. That's not very efficient, obviously, but we had to learn that. So it's most of the time, it's just one language per project, but that can change just depending on the people and the requirements. It's a bit easier to integrate Python in operative applications in our system landscape, and sometimes we have to resort to Java to be really compliant with all the regulations that we have in the industry.
I see Adam has a follow-up question here. Adam, you want to jump in? Yeah, I think that's interesting. I'm always thinking about that idea of polyglot workflows and how you mix languages, and that's something that I'm kind of passionate about because I feel like we did it mostly successfully in a previous role, but I know that there are lots of challenges that also make it kind of hard. I'm curious what your main challenges were and why you kind of are moving away from that.
Well, one of the challenges was package management and dependency management. So if you have both languages, you just have to manage more potential holes when you go to ITsec for clearance to go into production. So that's just a lot of work. It also has implications on the kind of documentation we have to do. It's just more documentation if you use many different languages. And it's also just the switching costs. So if one and the same person, even if he or she is fluent in both languages, just switches intraday between languages, people just felt that that is just a friction in their everyday working that isn't necessary.
In the beginning, it made sense because when we started off, we were like four, five, six people doing that. So one had a Python background and another had an R background, and we had to throw that all together in the first project. But as we grew, we had subgroups who focused on one of the languages, and then it was more natural to stay within that context in that one language per project. So maybe it's also a function of the size of the team. Yeah, that makes sense. I'm just a heavy R user in an increasingly Python world, so I'm always thinking about that. Thank you. Yeah, same for me. I mean, I feel most at home in R. The marketing mix model we're building, which I'm closest to in my data science work, is also completely in R. So that's my home zone. But I have to remind myself to stay fit in Python. We'll stay on this topic for a little bit because I see Alan had a follow-up here too.
Alan, want to jump in? Yeah, hi. Benedict, I'm curious how you support those new learners. Is there mentorship, guidance, et cetera? And I'm also curious if you get pushback from anybody to that sort of general expectation, like sort of challenging on the value, and how you sort of counter that. Have you got a really good value proposition that says, this is why it's really important to eventually become polyglot for your folks?
Mm-hmm. Okay, so in terms of the skilling process, we use DataCamp as a learning platform to do the data science trick there, just to have a harmonized set of skills and and that people just talk about the functions in the same way, things like that. So that's helpful just to streamline the actual coding. We have our own helper packages that interface with our infrastructure at various points in both languages. So everybody who joins just gets an introduction there, and then we'll have just the usual onboarding process within the projects that gets the introduction to the code and the specifics of the project. I don't have the feeling that I really get pushback, so maybe the pushback is more subtle and I just don't read it right.
Of course, we're joking about that. So there's the Python team, and then there's the R team, and we make our jokes about the one true language, but they're jokes in my opinion. They're not really, really serious. We had an initial phase where we had to discuss with the rest of the organization, especially, of course, with our T colleagues who are very Java-oriented. There is a new data science department who wants to do stuff in R, which is strange, and Python, which is not so strange, but still not on the radar, or hasn't been on the radar. That's a few years ago, of course, now it's standard. But that wasn't really pushback, but it was a new discussion and a more fundamental one. And you asked for the value proposition. I mean, at us, it's part of the interview and onboarding process. So we tell potential candidates very, very openly that we require R and Python skills, and it's also a plus just from screening students. So there is a clear expectation, everybody who joins has at least said that Rishi is fine with that.
Nice. Thank you. That's really, really interesting. Appreciate it.
Marketing mixed modeling
I see a few questions coming in around modeling, and I don't know, Saptarishi, do you want to kick us off with your question there on the marketing mixed model?
Sure, yeah. Hi. I just graduated recently with a master's in marketing analytics, and something that we just touched is marketing mixed modeling. I wanted to learn a bit more about what kind of tools you use for working with marketing mixed modeling, Benedikt. You said you use R primarily, but could you tell us a bit more about that?
Yeah, sure. So the starting point was Facebook's Robin model. So we have a mostly R set up there, and if you go through it line by line, you get a really good feeling about how to do the various parts that are required, like ad stocking and what are parts of a good marketing mixed model. Now, one of the things from this first experience with Robin that didn't work well for us was we had the requirement that we, on the one hand, wanted to know what the effects of our brand are on sales, but also we wanted to know what the effects of performance marketing, like search advertising, are on sales, and we wanted to have that in the same model. If you go to an agency and buy a marketing mixed model, you usually get two different models, one for the performance marketing and another maybe structural equation model for the brand part, because that's longer term. That was the initiator for us to build our own model, and one of the central requirements to have a marketing mixed model that we can refresh every day.
I mean, if the search colleagues come to me and have launched a new campaign, and a few days after that they come to me and want to know whether their campaign works, whether I see any differences in the marginal contributions of the different campaigns to sales, well, it's no good if I have to tell them, well, come back in half a year, because that's the refresh rate of the model, and then we have weekly data, which is the usual thing you do for marketing mixed model. That was one of the main requirements, and also one of the main challenges, because we needed daily data, not just from the standard APIs from Google and Meta, but also from our agencies, which had hundreds of sub-agencies and advertisers where they needed to gather their data, so that was quite a data engineering effort to get all that together, and then you have to do model adjustments.
For example, one of the central questions for the model are time-varying coefficients for the model, so if you use a standard marketing mixed model, you get one coefficient for one channel, that's what Robin does, for example, but if you want to do a daily model, and you want the model to pick up changes, then you can't have one coefficient for the whole time period, say a year, on a daily basis, because then you don't see any differences, so you want to see these coefficients change in the model, and there was a lot of experimenting with the feature engineering and how to get the GLM net parameters just about right, that these differences in coefficients come up in the model.
There are lots of other tinkerings and bits, for example, we played around with causal inference, because we had a lot of sales process data, and just threw that out of the window for the standard models with Robin, so we integrated a few different formulas and a few mediating models that represented the process steps in the causal inference chain that we had in theory, at least in our sales process, and then we can better isolate the effects to make that more ... to illustrate that in one of the first iterations of this model, we had a quite strong impact from social, from organic social posts on sales. Later on in the model, we implemented, we also added our internal email campaignings to the model, that weren't part of the initial data engineering effort and weren't there at the beginning, and then we saw from, while analyzing the causal inference chain of all these intermediate models and the chain, that the effect of the email campaigns controlled, when controlling for social posts on sales, was greater than the effect of social posts on sales controlled for email campaigns, which was an indicator that our social, organic social posts reached at least partly our existing customers, which still drove engagement, but didn't drive additional sales. So that's an, that was an interesting discussion with our marketing colleagues, how we could readjust the targeting of those posts, for example.
our social, organic social posts reached at least partly our existing customers, which still drove engagement, but didn't drive additional sales. So that's an, that was an interesting discussion with our marketing colleagues, how we could readjust the targeting of those posts, for example.
I was actually just going to ask you that, like how does some of the, how does the model change the actions of the team, like can you share an example of that?
Yeah, sure. So that was, that was the beginning of one of the examples. So in the development phase, we had this, this very short feedback loop where we would work on the model each day and just send two or one or two output plots of the contributions of different marketing campaigns to sales, just to our then head of marketing. And he would just, just via, via a team chat, and he would just give us very quick reactions like total crap. I don't understand this. This isn't, this isn't possible. How can we have negative sales there? Why do we have a contribution from something we do not spend any money on? So it also helped, helped us fix issues in the data engineering part. But it also just transferred to us a lot of domain knowledge about marketing that we didn't have. And also transferred hopefully to, to them a bit of how, what, how we looked at the data, what kind of insights we were trying to get. And after, I don't know, 50, 60, 70 of these exchanges, so it was for a time, it was just every day, the pictures started to synchronize. So we didn't, we, we, we had contributions from channels that they invested in. And we, we had one of few surprises and when they changed something, we could see something. And when we said, try this or that, and they tried, we could see a difference in the model. And that's, that helped a lot with the acceptance of those.
I'll echo what Alan said in the chat. It seems like you have a great partnership with the team. Yeah, definitely. I mean, we've been working with that team and with a lot of other teams as well, but with that specific team for the last seven years, wasn't, wasn't the first effort to build a marketing model, a marketing mix model. And the first efforts didn't go as well. But we finally found an approach that worked. Yeah.
Model management and governance
Sam, I see you have your hand raised. Do you want to jump in here next?
Yes. So my question was quite long. I'm trying to simplify it in a way that's more tangible. So I work for a health authority that runs about seven hospitals in BC, Canada. And right now, our senior executives are thinking about an AI platform. And so the complexity is we have a bunch of models that we're probably going to, and it's a small piece of it, right? Where it's going to be handled, tracked, and they can be either private providers, custom built, or they can be private providers, custom built into in-house. And the question is, how are you managing them? Like we, I'm recommending Weights & Bias, MLflow, or Neptune AI, some kind of already available framework and use that to manage the custom models or vendor models. The other question is, you know, there are so many models coming in from various departments internally in your organization. Like, how do you know what's what and how do you keep track? Do you guys do it your way or you have these third-party providers? That's my question. Sorry. Thanks.
I wouldn't say there's one way to solve all these problems. In a way, it depends. So what we do to manage the model in a way depends on the application or the area of application. So there are models that just are meant as decision support or informing decisions. And for those models, the regulatory requirements are not as high as for other models. So we try to keep the regulation overhead low while still fulfilling all the requirements that we got, which are still enough. But for those models, usually it's enough to have the code versions, to have version tracking for the different model iterations. And just, for example, upload those models into GitLab and use a GitLab runner and have a schedule there. So that's one of the simple cases.
For the more critical applications, MLflow is one of the alternatives that we use. I would assume that we will use it in the future even more than we do now with new incoming regulation coming from the AI Act. So that we also need to take, for example, all the experiments and all of this. And for the most critical applications, we, for example, when we build models that help with pricing, they are integrated into the production pipeline from our pricing software, which is specialized insurance software. So we adapt to that format. And you have, of course, to invest time and resources to find a good way to, for example, extract the information from your model via PMML or something like that, or even cleverer formats and get that into a proprietary software suite. We don't rely that heavily on third-party models.
Well, it changes a bit with LLMs, but that's a special case right now. Apart from that, we don't really rely on third-party models. One reason is governance and trustworthiness. So we want to have
the full control over what the model does. Of course, also the full responsibility, ensuring that they are compliant. But also, I'm really skeptical about vendor lock-in and possible cost cuts that will then disable you from building new use cases. So for me, it also has to do with this robustness to be able to help yourself and build use cases in the future, not dependent on one single tool and maybe one discussion or management decision from one tool to another. So I don't want to be dependent on a proprietary tool.
Building and structuring a data team
Thank you. I see, John, you had two questions. So maybe start with whichever one you want to ask here first. No, I appreciate it. Maybe I can merge them into one. So I'm starting a data team myself. I'm hiring up a few people. And I was wondering if you had some perspective, since your team is obviously much, much larger, on just how to set up the team itself, right? So things like, are you having some people specialize in interfacing with the business, like with marketing? Or is it everyone's doing that, or some people sitting in the corner doing more of the hardcore analytics? So I'm just thinking about the titles and the structure and the organization design. I was hoping you could give a few insights.
Yeah, sure. Maybe I start with one or two principles we have, and then the structure will fall into place. So I said at the beginning that I think it's important to keep things as interesting as possible for the individual data scientists. So one of our principles was, anybody is, in principle, allowed to do anything, to work on anything. So I don't want to have a setup that prohibits anybody to look into a topic or a potential model or use case that he or she finds interesting, or acquire a skill, a technical skill, that he or she finds relevant and interesting.
The other principle is that the best use cases start from business questions. So I do not want to be merely a business support function. I want to be part of the business. I want to incentivize everybody in the teams to not just build the model, to have the attitude, give me the problem and I'll build you the model. So that's not the right attitude. You have to want people to solve the business problem so that they are really part of the business. It also gets you a seat at the table when the business decision comes along. So that's a benefit of that approach.
So with that in mind, we had our team set up along the axis for whom are we working. So we have one team that focuses on gaining insights for human consumption. And we have another team that's more focused on building models. And the third team that focuses on applications that use ML models. Naturally, the first of these teams, so the insights focus team, is closer to departments like marketing or various other business functions. Or the last of these three teams is closer to an IT function. But it helps us also to avoid handoffs, for example, to IT or handoffs between the teams, because we can just create virtual teams between these three teams for any task that comes up. So it feels a bit complex, but it forces everybody to work across teams all the time, which I believe is a huge benefit.
Mixed models, stability, and retraining
Absolutely. Love that. Going back to the mixed model for a second here, I see Renhua, you had a question. Want to jump in? I'm Renhua, I'm a data scientist in Calgary. And I have a question. I have a background for SAS and R and Python. And I tested the operation for the mixed model. And I think R is more like I have some kind of package very similar to SAS. And Python is, I think, is a I try to do this through the R, Python to be able to load in through the R to run the model. And I found that when I use R or Python, I think, how to say, they have some string testing that SAS is missing in R. So I think, what do you think about this kind of platform? And what do you think about stability after you build a model and how stable it is?
Okay, that's a good question. Now, if I work on the model, I have to be really careful. So we have implemented a lot of tests, because there are parts of the model that can make the model really unstable. And it's quite hard for me to predict where this instability occurs. I think that over time, I've gotten an intuition about that. But still, if you, for example, play around with the lambda parameter of GLM net, and you just have the bad luck to get the wrong three lambda parameters, then the model output is totally different. To avoid that, one of the strategies to avoid that is that we do not really build a single model, but a whole host of models and have them consent on one view, how the channels are broken up and how we calculate the different contributions.
And one of the other mechanisms that I like is that we use the marketing spendings in the past from our marketing colleagues as a guidance to what the most plausible model is. So we have the baseline model, which is just only looking at the data, not at how marketing thinks the contributions should be. But then within the distance from this best performing model, when it comes to RMSAE or the metrics, we search for a model that is most plausible, given the expertise of our marketers, that is also represented in how they spent their budgets in the past. And this also helps, of course, with the acceptance of the model.
And this also helps, of course, with the acceptance of the model.
EDA workflows and model retraining frequency
Batul, I see you had a question that might be a follow-up here. Do you want to jump in next? Sure. Hello, everyone. I'm a data scientist at John Deere. So the two questions that I had is, how does your team, specifically individuals coming with native exploratory data analytics experience using R, transition into doing that with Python? As I do machine learning models myself and try to explore data, I feel I need to go to R, do my exploratory data analysis before I build models with Python or PySpark. So what's the strategy to easily transition to Python and do that easily? Second question is, going back to your question on the model coefficients, how often should a model be updated to capture the changes if we know that the data is going to be changing continuously?
I don't feel that people in our teams really have to switch between R and Python when they go from EDA to building the model. But I see differences in how they do EDA, depending on whether they use R or Python. So if I have an R user, I probably get an R notebook or Quarto notebook with a few ggplots, maybe not so much interactivity and rarely get a Shiny dashboard for IDA. So maybe that's then people just moving on to Superset and doing the stuff there. If I have, if I see an EDA in Python, it's more likely to be interactive using Dash or maybe it's even demo-like as a streamlined app. So the starting point is probably more closer to the actual development.
Since we stay within one language for the project, we don't switch within the project. But maybe there's a design part of our engineering or development cycle that helps there, because most of our use cases are created after a manual analysis. So most of the time we have just a business question that pops up when we do reporting or just manual analysis. We go into that question using proprietary tools like Google Analytics or the tracking solution that we have, which is a different one, or SAS maybe, and partially R there. And then when we discuss whether this can become a model or a use case, and we do the business, the cost-benefit ratio and all the analysis we have to do before we actually start a use case, then we have a different discussion of whether we keep it in R or do it in Python. And that takes away this switching pain from EDA to model development, a bit at least.
Second question again was... It was about how often should the model be trained or retrained or updated to capture changes? Yeah. So the model is built with the time-varying coefficients in a way that it can adapt to changes itself. So we don't have to retrain everything all the time. As a matter of fact, we do it roughly once a week, I would say. So every time a change in the variables comes up, that triggers the tests and gives us a heads-up that we have something out of the standard deviation, we should consider retraining. So usually we do it then because it's not very expensive. Okay. Thank you.
Being part of the business, not just support
I'd love to see the questions to everybody in the chat as well. I saw Amiko asked a question to everybody if you want to weigh in, but tips on how you convince to be a part of the business instead of support. And so I see a thread there starting, but Benedikt, do you have anything you would want to add to that or tips on being a part of the business instead of support?
So one of the things I see, particularly in people who are new to, who come newly into the team and aren't, maybe even from the industry, what happens a lot if you build dashboards or reports without industry experience is that you talk about the what that you see in the data. So you have, in principle, somebody reading out a dashboard to you. That may be interesting and the dashboard may look nice, but that's not really interesting for the business. They can, most of the time, they are quite quantitative. So they see the what immediately and most of the time, even better than I would, for example. But the why is actually the important question and you don't normally get to the why if you just look at the data, you have to get out of your data science home zone area in the office and just go to the people who do the real work.
And you come most of the time back with quite a few number of hypotheses about what's going wrong or what's going right about that. Of course, you can then think about designing experiments. But most of the time, if you just dig deep into the data, you can find a quite obvious answer that part of the process is just maybe broken or just faster than another process.
So this obsession with getting to the why and getting your hands dirty in non-data science work, I think that's most important to be part of the business because everybody else in the business does it as well. You're not in a better position or you're not allowed to have any kind of attitude just because you're a data scientist.
So this obsession with getting to the why and getting your hands dirty in non-data science work, I think that's most important to be part of the business because everybody else in the business does it as well. You're not in a better position or you're not allowed to have any kind of attitude just because you're a data scientist.
Thank you so much. A question I wanted to ask you and I think I see a few questions coming in about AI, but I haven't explicitly seen AI called out in as many head of analytics titles yet. And so I was wondering if you could just share a little bit more about that and the work that you're doing with AI. Sure. So one of the focus areas of right now is, of course, generative AI.
And it's part of, well, it was borne out of last year's hype about that kind of artificial intelligence. We are currently, for example, working on a system that helps us retrieve knowledge from these very old insurance policies, which our customer agents spent quite some time on.
We have various, very complex and partially very old knowledge databases where our customer agents have to gather multiple information across, I don't know, five databases to answer a question if it's really, really complicated. And they need then maybe even two or three other colleagues to be really precise about the answer. Because if you are not precise, then the company is, of course, liable for every false statement that the customer agent makes.
And we are currently in the process of building a retrieval augmented generation stack to help get to the right documents faster. We're also experimenting with generating the answer. We have to be very cautious about that because of the liability issues.
We are currently also very careful not to expose that to the end customer. I want to have a competent human always in between to check that. It's still in the development phase. But that's one of the main use cases where we use LLMs and general AI.
Thank you. Abigail, I think you have a follow-up on this specific question. You want to jump in? Yeah, thanks. So I guess what is your architectural look broadly for your RAG chatbots and what are you doing for evaluation versus for formal assessments?
Mm-hmm. So we use quite a standard RAG architecture with an external API call to open AI at the moment, but we're evaluating switching models there. We are estimating four metrics, if I remember correctly, at the same time. So factual correctness is one, relevance, conciseness, and completeness is the third, which I always find the hardest when it comes to completeness. Because insurance policies for our flagship product are 277 pages for one product. And it's hard to find out just how much text you quote and how much options and writers and exclusions you add to the answer you want to generate without overloading the whole chat. And it would be useless. People could just read all the pages and they wouldn't speed up in their operative process. So it's hard to get that right. And it varies with the stakeholder you're talking to. That's a different kind of challenge, but it's also a challenge if I talk to custom agents. They want different things than if I talk to product experts. And maybe it would be best to switch interfaces. I don't know yet.
Gen AI and predictive models
Thank you. Maximilian, I just want to make sure that that also covered your question. Yeah, I think so. Maybe just a follow-up question on that. Greetings from Munich, by the way. Did you also think about gen AI in combination with your predictive model? So you work a lot with marketing models, next best action, next best offer. And did you think about using them in order to maybe explain the output a bit better or creating content around it?
Hmm. Um, I haven't really, really invested deeply into that. So when we, we are thinking about one use case where we create content for campaigns. Um, I haven't really tried out creating explanations for the models. Um, maybe I should give it a go. Good thought.
Causal inference in practice
Awesome. Um, Rick, I see your hands raised. Want to jump in here next? Thank you, Rachel. Good to see you again. And Benedict, thank you. Uh, by the way, in the chat, spellcheck stumped on your name. I apologize on behalf of the rudimentary AI behind spellcheck.
Uh, your remarks about getting to why are brilliant and, uh, personally, I'm a fan of causal inference. I'm wondering to what degree, and you also mentioned causal inference earlier in your remark. Would you please elaborate on the extent to which you and your team, uh, employ causal inference in your SD work? Um, I'm sorry, your DS work. Does your team embrace professor Pearl's the book of why or professor Hernon's what if book, uh, from Harvard. And to what degree do you, are you and your team trained in causal inference and to what degree do your business partners know what you're talking about when you discuss causal inference, particularly as a path to get to that so important question of why your thoughts, Benedict.
Yeah. Okay. So that's, that's quite a lot. Um, so I do embrace Judea Pearl's book of why, um, and I try to, uh, spread that in the team. Um, but that's still a process. So we are still in the learning phase there. So, uh, nobody in the team has received formal training in causal inference as far as I am aware. And maybe my team will tell me afterwards that I'm wrong. Um, but we are trying to educate ourselves as a good as we can. Um, since we have done that in the past quite successfully, um, I believe, um, we have a good chance that, um, I'll, I'll just elaborate how we, how we use causal inference in the model, uh, where we use it. And I think then it's easier to answer whether the business partners are aware.
So we use causal inference, um, mainly within our marketing mix model that I've talked about a bit. Um, the, the, uh, reason to try out causal inference there was that we have a lot of data about the sales process that gets normally thrown out. Um, when you do standard econometric marketing mix modeling. So we, we usually, you have the marketing spendings for various channels and try to models sales as a function of the marketing spendings. And then you modify the spending standard marketing mix model, but we have so much more data. We have data on the impressions are estimated on, on the clicks, on the wizards, on the calls, on the email campaigns. And we have that quite on a granular level. And I always thought that it would be a waste of all this information and signals, um, just to throw it out in an effort to do a marketing model.
So we started, uh, drawing a causal diagrams where these process data are mediating variables. So we have our independent variables, the spendings, and then we have a lot of mediating variables. I mean, we wouldn't do any quotes if we didn't have the spendings in the first place. And we have to do a quote before we can, um, sell an insurance application, sell an insurance policy. Um, it also, uh, helped with controlling for unknown confounders, because that was one of the issues that early marketing mix models didn't get off the ground. You know, one of the best ways to, um, to sabotage any modeling effort of anybody is just ask, uh, just think of one of one potential confounder and ask them if they have considered that confounders provide discussions about whether I've integrated weather, unemployment rates, uh, stuff like that. And many marketing mix models from agencies do.
So I wanted to find a more elegant way around the problem. And so now what we're doing is, um, we use, we built, uh, models, several models from spendings to these mediating variables. And then we built one model from the meeting, mediating variables to sales. And then we can do, um, a call. I think it's front door adjustment, the front door adjustment in the, in the causal inference, uh, from spendings to say it's induced that simulations. So our business partners, I don't think they, they are deeply interested in the specifics of causal inference. I have to say, but they are very interested in the causal diagram because it represents how they see they are the process that they are. Uh, and I can offer a better, better basis with them for connecting and discussing how this process works and how the model works. And we just displayed the causal diagram.
but they are very interested in the causal diagram because it represents how they see they are the process that they are. Uh, and I can offer a better, better basis with them for connecting and discussing how this process works and how the model works.
Career advice and closing
Thank you. So I realized we are almost to the top of the hour here. And I, I think at some point I will pull all the answers to this question together from all the leaders, but I love getting to ask about a piece of career advice that you've either received or given that you would like to share with us. Okay. Um, one of the earliest pieces of advice was Benedict, you're going to fail. It's, it's a guarantee. You're going to fail. And maybe you're even going to fail every single day. You just can choose where you fail.
So to elaborate a bit every day, you receive tons of deadlines and emails and requests. And it's, it's a certainty that I cannot meet all of the things that are thrown in, into are thrown in, into my direction. And that's fine. I mean, that's part of my job. What I'm, I'm not thinking about trying to meet all the deadlines and do all the stuff that I have to do. Um, I'd rather like to think about where, where can I afford and where do I want to fail and not meet the expectations that has helped a lot with accepting things that go wrong and discussing it more openly with my team.
You're going to fail. And maybe you're even going to fail every single day. You just can choose where you fail.
Thank you so much. That's always a big one. How do you figure out what to prioritize? Um, we have, I guess, just a minute left here. I just want to make sure if there were certain questions I had missed, if anybody wants to raise your hand or recopy something over, I'll give us a few seconds here.
You just sent me a question on LinkedIn if you want. Okay. Yeah. Is that the best place to connect with you? LinkedIn? I think so. Yeah.
Okay. Perfect. Thank you so much, Benedict, for taking the time to join us and sharing your, your insights and experience. This has been great. It was a pleasure. Thank you. I appreciate all the, all the great questions and comments in the chat as well. Have a great rest of the day, everybody.