Resources
Feb 7, 2024 |
59 min
|
420 views

Mehran Moghtadai @ TD Bank Group | Data Science Hangout

We were recently joined by Mehran Moghtadai, AI/ML Accelerator & Enablement at TD Bank Group to chat about working to enable data & machine learning scientists to focus on delivering high-value use cases with as little friction as possible.

Featured Leader Bio:
Mehran Moghtadai is a Senior Manager at TD Bank leading the AI/ML accelerator and enablement team. Leading the creation and maintenance of numerous internally developed tools. Mehran joined TD Insurance as an actuarial analyst in 2014 working on various modeling projects as well as the implementation of the first optimized pricing algorithm at TD. Mehran holds a Masters in actuarial mathematics and finance from Concordia University.

________________________

ā–ŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software

To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)

Thanks for hanging out with us! šŸ’›

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi everybody. Happy New Year, if I haven't said it yet to you, and welcome back to the Data Science Hangout. I'm Rachel. I lead Customer Marketing at Posit. I'm excited to be back here with you all. This is our first Hangout of 2024. The Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. We get together here every Thursday at the same time, same place.

So if this is your first time joining us, it is so nice to meet you. Is it anybody's first Hangout? Say hi in the chat, as we'd love to welcome you in and say hello to anybody joining for the first time too. We're all dedicated to keeping this a friendly and welcoming space for everyone, so we love hearing from you no matter your years of experience, titles, industry, or languages that you work in. It's also totally okay if you just want to listen in here, and awesome to be a part of the party that happens in the Zoom chat too. But there's also three ways you can jump in and ask questions today or provide your own perspective on certain topics too.

So if you've been here before, you've heard this whole spiel before, but the three ways to jump in are you can raise your hand on Zoom, and I'll call on you. You can put questions in the Zoom chat, and feel free to just put a little star next to it if it's something you want me to read out loud instead. And then we also have a Slido link where you can ask questions anonymously, and our team will share that in the chat here in just a second. A quick note that if you are watching the recording at some point in the future on YouTube and want to join us live, the link to add it to your calendar will be in the details below to join us. And there's no rule that you have to stay the whole time or talk. You can come and go as if it's your schedule.

But with all that, welcome. I am so excited to be joined by my co-host for today, Mehran, AI and ML Accelerator and Enablement at TD Bank Group. And Mehran, I would love to have you introduce yourself too and share a little bit about your role and also something you like to do outside of work. And you can also educate us on how to pronounce your last name as well, because I know we didn't get to that in the first part.

Yeah, no worries. Yeah, so it's the proper way is Mehran Moqtadaei. But, you know, I've been in Canada since I was like six. And I think I accept many different pronunciations. So you know, that's not a problem. So yeah, so super happy to be here and hopefully get to chat with some of you folks. Yeah, so I've been at TD Bank Group for a while now. So it's coming close to 10 years. So I started as an intern there, actually. And my background is actually actuarial mathematics. So I got there and, you know, I want to be like a very good actuary, do all my exams and everything. But since then, I kind of deviated quite a bit, I would say. So I had some non-traditional roles there, some innovation teams, and then sort of like looped back into sort of more the data science world. And now I lead like a small team of sort of, it's fairly, I would say, diverse. There's some actuaries in there, some more ML engineers and ML scientists. And the goal is sort of to, you know, get these sort of use cases started and across the line, where in some industries, you know, there are more challenges and we can, I'm sure we're going to be talking about some of those.

On my sort of like free time, actually, this holiday season, I got really into like home automation stuff. So, you know, I'm a lazy person, but I'm also the type of person that will spend like, you know, 10 hours to automate something that probably takes like, you know, a few minutes of like manual work. So it's a bit of a, you know, counterintuitive thing. So, yeah. And I have a friend who is very into privacy and those two things don't go well together. So home automation and privacy, typically you have to subscribe to a bunch of cloud services. So I got into this thing called Home Assistant, you have your own server. So it kind of took a lot of my time. So that's, you know, if you want to talk to me about like, you know, home automation and server management and things like that, you know, feel free. That's sort of like what I'm into right now.

Okay. I need to know what one of those home automations is then. Okay. So I live in Montreal, so in Canada, so it gets pretty cold. And my house, I have like a home humidifier, like a whole home humidifier. But the thing is, you know, when the temperature drops below a certain amount, and if you set your humidity levels too high, your windows start getting foggy. So mine is like manual. I have to go down in my basement and like manually adjust it so that it doesn't get foggy. So this whole setup, I have like humidity sensors and with the external temperature sensor so that it automatically adjusts so that I don't have to do this manually. So that's, it sounds straightforward, but that sort of thing actually took a decent amount of my time to set up, but it was cool. It was fun.

Introducing use cases and the accelerator mindset

Awesome. I love hearing about everybody's side projects here. Well, while I wait for questions to come in from everybody here, I thought to get us started, could you share an example of maybe a data project or a use case that your team has worked on? I think that's always, it always helps us kind of better understand your day-to-day work and challenges. Yeah, for sure. So there's some particularly interesting use cases that we've worked on. And typically, you know, our use cases always start from let's say a problem statement, right? So we try to position ourselves as not being sort of like a, you know, our only tool is let's say a model and that's what we're going to deliver to you. You know, we look at the problem, we really try to discuss with the business what they need and sometimes, you know, the answer might be, you know, you need to talk to a vendor, right? Like this isn't something we want to do internally. And sometimes it turns out that if it is something that we want to do internally.

One of the use cases that was particularly really interesting was, you know, this is before COVID times and then during COVID times where our call center started getting some operational issues. And when COVID happened, things just started getting worse and worse. And, you know, a lot of folks tried to come up with solutions and approaches to kind of solving some of these problems. But what we really realized very quickly is that we don't have the proper, you know, data set curated to actually even understand what the problem is, right? Like, and then try to give solutions. So that was a huge amount of work. And initially, it started out as like building a curated data set, but then it evolved into actually like a full-fledged coaching application for our call centers. And this is something that is ongoing today. And it's used quite a bit where we give, you know, not just, you know, metrics and things, but actually we tell the managers, like, this is specifically what you need to look at. And this is what the specific person needs to be coached on. So just being able to give those insights directly to people and, you know, there's a front end to it. And, you know, Posit has been like extremely helpful in that, but the amount of like data and, you know, sort of like models and things that go behind that, and it's all transparent to the end user, but that's one of the really cool projects that we got to work on.

That's an awesome use case. Thank you for sharing that. So what does the front end look like for them or what is the front end? Yeah, and that's the other thing. So one of the other things that we were able to do, and this was sort of like part of the accelerator mindset in our team, I guess, is that, you know, we work really in MVP. So, you know, Agile is the, you know, flavor of the month, or at least it's been the flavor of the month for a while now. But like, we really kind of like, you know, take that to heart in the sense that when it started off, the UI was fairly straightforward. It was a sort of like a Shiny application. It gave some insights, some things. But I would say we did like feature releases on a monthly basis. And even earlier on, it was even faster. We started with a pilot program where we had, you know, a group of advisors, and they kind of like grew and became fairly large to the point that, you know, it was all of the insurance business was covered under this. So it started as a Shiny application. Then, you know, we realized that it's great, things are good, but if we want to sort of like scale it out like even further, now that it can't be done with Shiny and you can definitely, you know, do things, modules or things like that, we had the resources available to us to kind of take it a step further and like create like a proper back-end set of APIs and create like a front-end in like JavaScript with Vue.js. And that's how the application is today. And it's still evolving. Like it's still, you know, there's monthly feature additions with the feedback from the business, but it's sort of like Agile way. That's sort of the, I would say the main benefit of something like this.

Speed of reaction to business needs

Thank you. I know when we were talking before, I guess this kind of, this continues on this topic, but on our pre-call you talked about like the speed of reaction to business needs. And I was just wondering how has the emphasis on the ways you interact with business users and that speed of reaction contributed to your team's success? And you can speak in terms of that use case specifically or a different one too. Yeah, I would say that is the main sort of ingredient of success because at the end of the day, so there's maybe two elements, but that one is very important because at the end of the day, you know, a lot of times and, you know, business users, but just typically as a person, when you think of, you know, oh, this is something that I want, you might say it, but then when it gets into your hands, you realize, okay, like this is what I said, but actually now I realized what I wanted is something different. And there's two approaches to this. One of the approach is, you know, you sit down, you do a lot of like, you know, workshops and you try to create like a proper requirements document. And that's, you know, for certain uses that are very like high risk and you need to make sure you get it right on the first try, that is appropriate. But for a lot of uses, it's better to just, you know, give the user something so that they can use it and give you feedback. And then based off of that feedback, again, you have to kind of, you know, take it and understand, okay, they're giving this feedback, what does it exactly mean? And give them something that kind of addresses that. Being able to do that very quickly, that is a huge element of the success here. Because after a while, we started having this relationship with the business where they knew that, you know, what we gave them, if it wasn't perfect, they could come back to us and we would take it a step further, right? And having that sort of like trust, that's what kind of enables a lot of these projects to kind of get off the ground.

Because after a while, we started having this relationship with the business where they knew that, you know, what we gave them, if it wasn't perfect, they could come back to us and we would take it a step further, right? And having that sort of like trust, that's what kind of enables a lot of these projects to kind of get off the ground.

Sustaining innovation: support, ownership, and IT collaboration

Thank you. Alan, I see you have a question in the chat. You want to ask that here? Yeah, sure. Hi, Moran. I'm really curious. I've seen tension in the past with like accelerator or innovation teams making really cool things and then running into problems of like, how do those get supported long term? Because like these little innovation teams usually aren't resourced to handle that. Do you run into that kind of tension? And how have you figured out how to like make things that are sustainable and know when to hand them off and know that you've got partners who can take them? Any good solutions in that kind of organizational area? Yeah, for sure. So this is this is a very important point. And I think that's been one of the biggest challenges that we've solved sort of incrementally. So one of the most important things is even before starting a use case like this. And I think, you know, it requires a certain level of maturity that I would say like early on in our journey, we maybe didn't have, but we were kind of lucky in certain ways, is you need to be very direct with the business and make them understand that like, look, we can build this, we can build this very quickly. And it's fun to do to build, but it will require maintenance and maintenance means there is costs associated to that. So we need to make sure that whatever they're asking for has benefits, right, like actual tangible benefits, so that when we do go back to them and say, like, look, I need to, you know, assign like an FTE to kind of do long term support for this. They're okay with funding that, right. So that part is very important. And, you know, we need to be very direct about that very upfront, right, which is why a lot of the use cases that we tend to support, they're fairly high value. And when they're fairly high value, you go back to them, you're like, look, I can build this, it'll cost you next to nothing. For support, we need to kind of support this, and that will kind of have some costs. Typically, they're okay with that.

So that's one part of it. The other part of it is, you know, making things sort of standardized. And when I say standardized, I don't necessarily mean that, you know, make like a cookie cutter application that kind of applies everywhere, but have these sort of like frameworks that, you know, you customize that, you know, if you take somebody that doesn't have experience in this, but he's worked on a similar application, he can come and he can support it as well. So by doing that, we're able to have like a fairly small group of fairly small group of people that can be the sort of ongoing support team as well as the development team. So that's part of the equation. Then there's another part of the equation is that, you know, you start kind of looking like an IT team once you start building all these layers, and we're not an IT team, right? So then you need to have a proper relationship with your IT team and kind of, you know, set the limits of like, okay, well, the infrastructure you own, you own sort of like, let's say even the deployment of the applications, but like the L2 support, like if there's something wrong with the application and stuff, then that's what we own. So making sure that you have these operating models that kind of make this work, that's sort of like the magic recipe, because if it's all on the, let's say the analytics team side, that's not really going to be sustainable and work long-term. And if it's all on the IT side, you know, we've seen that that doesn't work either. So having this collaboration is really the key.

Nice. Yeah. A lot of that sounds familiar. We're still trying to figure out organizationally how to solve some of those things, get support for the things that we want to live in IT versus not,

and it's tricky. It's very tricky. It's great to hear you've got a successful model. It's really cool. Thanks. How did you set up those different levels for like, who handles which applications or what is considered IT? Yeah, that's a good question. And, you know, it's a tricky one. And I feel like our story is probably not that different from a lot of other people. So,

you know, initially, this kind of started off as something that's like, you know, kind of, you know, back of the envelope, and then we got something handed to the business, and it kind of like started growing incrementally. We had the necessary resources to make sure that from the business's perspective, we were able to support them. So, and the fact is, because these applications kind of always get new features and get everything, they're never really in maintenance mode. It's not like, oh, it's set, and we just need somebody to like, make sure it runs. They're always in development kind of, right? So, because of that, we always had the resources to kind of support this. And the business, you know, from their perspective, they were supportive. So, they were super happy. We got to a point where then the enterprise was like, okay, you guys are doing things that are like very operational, and you're kind of doing things that like IT should be doing.

And when we got to that point, it was like, yes, we kind of agree with you, but we had already made the business case that there's value in doing this this way. Because I would say before we even got there, the discussions were like, well, no, this, like, you guys shouldn't be coding. It should be IT that does it, right? But once the value was there, and once the business was like, wait, I like what these guys are doing, then the discussion kind of changed, and it became more like, okay, well, you know, you have the analytics team that is building these things, but like, how do we put all the right controls and everything in place to make this something that is sustainable and is within the risk profiles that, let's say, a bank like TD kind of accepts, right? And that is really, you know, you need to bring in a lot of partners. It's a lot of discussions, and I would say these discussions kind of take like years, right? It's not something that gets solved like, you know, in a single meeting. There's a lot of back and forth until you kind of find this middle ground of an operating model where you keep your agility, but you're able to have all the right controls in place to make sure that, you know, everybody's kind of, you know, feel safe and happy.

So, would you recommend building something impactful for the business first before thinking too much about like who handles what? So, I mean, it depends on your industry, and it depends on how conservative your industry is, right? So, the banking industry in Canada especially, right? Like, Canada, you have like six banks, and that's it. So, it's very conservative. So, the risk of doing something and hurting the bank are almost always higher than the risk of not doing anything at all, right? So, in that kind of an industry, you have to be very careful, and the truth is, like, you know, we do handle a lot of very sensitive information. We do have to be very careful with these things, right? Like, that is priority number one whenever we're doing any project.

So, in these kind of industries, having something that is tangible that you can get like a small group of users to use it and see the benefits goes a long way. Now, how far do you want to take it before, you know, you want to have those conversations? That kind of depends on, you know, how risky that application or whatever you're building is, right? Like, if it's something that requires, like, really high risk data, it's very operational, then you might want to have those conversations very early on. But if it's things that, you know, they're very low impact, let's say the business can live it without if it goes down for, like, two weeks, then, you know, you have a little bit more freedom. But you do need to build that trust with the business, and the business does need to see, like, something tangible that they can, like, kind of understand and touch, because that's when the value becomes kind of, like, real.

But you do need to build that trust with the business, and the business does need to see, like, something tangible that they can, like, kind of understand and touch, because that's when the value becomes kind of, like, real.

Tools and technology stack

Thank you. I know some people join right after the beginning where I give the whole spiel about how you can ask questions. So, I want to remind everybody that you can just raise your hand here on Zoom, and I'll kind of keep an eye out. If you want to jump in, you can put questions into the Zoom chat. And if it's something you wanted me to read out loud, just put another star next to it, too. Maybe you're in a coffee shop or something. And then we do have the Slido link I just reposted in the chat, where you can ask things anonymously, too. I realize if I say that in the first minute, some people may have missed that.

But Maren, I'm curious if you can share a little bit with us about some of the tools that your team uses, or maybe some insights into, like, key tools and technologies that play a significant role in your workflows. Yeah, for sure. So,

we are actually mid-migration. So, currently, our main analytics platform is a Hadoop enterprise platform, like, self-hosted on-premise. So, we have tools like Spark. Spark is sort of, like, the main go-to hive for, like, data ETLs and things like that. That's sort of, like, on the infrastructure side. More sort of on the data scientist side. So, we do use Posit Workbench quite a bit on-prem. Our insurance business, in particular, uses Posit Workbench. The rest of the bank tends to use JupyterLab quite a bit. We have sort of, like, containerized JupyterLab environment. And we have Posit Connect, which is where all these sort of, like, applications on front ends kind of end up living, is Posit Connect.

Now, moving on to Azure. So, we're making a big migration. We're very Databricks-centric at the moment. We're looking into other tools like Azure ML and things like that. But today, it's very Databricks-centric. So, there's a whole stack of tools there, but Databricks being the main sort of, you know, analytics engine environment. So, again, more Spark-heavy.

We're looking to migrate everything that we have on Posit Connect onto a new instance of Posit Connect in Azure. So, everything will kind of get brought there. And then for our machine learning models and everything, they'll live on Databricks for now, but we're looking, like I said, into other sort of more flexible tools as well.

Thank you. I appreciate it. I see Lisa's comments, and I think it was at a fairly techie conference I went to where someone said, organizations are always mid-migration.

Abigail, I see you had a question in the chat. Do you want to jump in here? This is, like, a very specific question. So, I know there's, like, so Databricks, obviously, gets the notebook environment, which, you know, some good things, some bad things. I know they, on paper, have this, like, VS Code connector or something like that, but I've never met anybody who's actually using it. Are you using it? Can you speak to that? Thank you. Yeah, that's a good question. So,

we're not using it. I wish we were. But the reasons are more because of particularities of the way that TD has sort of, like, set up the environment. So, Databricks, it's fairly flexible. Like, they have a good set of APIs that you can kind of plug into. So, personally, I'm not a huge fan of notebooks. I would much prefer using, like, a proper ID. But connecting to these APIs has some requirements in terms of, like, path tokens and things like that, and then that has, like, security considerations where it's not opened up for us. So, we still do have certain gaps, which is why I'll do a plug for Posit here, why we're kind of interested in this new engagement that Posit has with Databricks of integrating Workbench into Databricks, because for us, it kind of solves a lot of issues because we have one single managed environment that we get access to, and then there we're, like, already authenticated into Databricks and we can kind of use VS Code and stuff. So, that's sort of, like, hope. But, yeah, we haven't used that external connector as of yet, just because we don't have that, like, external connectivity set up. Okay, thank you.

Data science team structure at TD Bank

Thank you. I know you covered this a little bit about your team, but could you tell us a little bit more about, like, the data science team structure at TD Bank? Yes. So, TD Bank...

So, TD Bank has a lot of different lines of businesses. So, you have these various lines of businesses that each have their own sort of, like, leadership. All of the enterprise data and analytics is essentially under a single leadership, and we have different what we call, like, practices under these leaderships. So, you know, we have a BI practice, we have a sort of, like, a data engineering practice, and we have a AIML practice. So, we reside in the AIML practice, but I would say we're very well integrated. So, we have very close connections with the line of businesses, even though organizationally we might be sort of, like, all grouped. Let's say, like, all the lines of businesses, the ML, AI practitioners, they're all grouped under one, let's say, VP, but we all work very closely with our, you know, respective lines of businesses. So, that's roughly the structure.

Do you, like, get together with, like, community initiatives or, like, annual conferences or things like that, too? Yeah, for sure. So, we have town halls, and again, depending on, you know, we have, like, three different town halls depending on who is included. So, we have our own sort of, like, the AIML practice, and then we have the larger enterprise data and analytics family, and then you have the different town halls with the lines of businesses. So, we do have these, so we do have these, like, large conferences. And I mean, you know, the enterprise data and analytics community at TD is very large. I would say probably we're talking about, you know, 4,000 or so data practitioners. So, this includes the AIML practitioners and everything.

So, yeah. So, it's a very large group of people. So, we have, like, subgroups that we need. We have a larger group that we need. So, it includes a fairly significant number of people.

Open source adoption in a regulated industry

People are a little bit shy today, I think, coming back in the new year. I know there's not as many questions in the chat, but I get to just keep asking Mehran as many questions as I want here. But Greg, I see your hands raised. Do you want to jump in? Yeah, absolutely. I work for Charles Schwab. So, I feel like we're, like, cousins or something. I don't know what the relationship would be, but there's a history there at TD and Schwab. So, one of the challenges that I face here, I've been here for six years or so, is kind of an aversion to open source packages, products, anything that's not kind of vetted at enterprise, purely from, like, a risk perspective. I've had so many conversations about, you know, even well-vetted packages like dplyr or pandas, and, you know, how can we guarantee that there's no malicious code that's going to, you know, come scrape and steal our clients. And as you mentioned, it's a function of being in a very highly regulated industry. Have you run into that? And how have you, like, maybe kind of put those fears aside and kind of, you know, confirm that these are really safe and well-vetted tools in a variety of industries?

Yeah, that's a good question, because I would say, so, 10 years ago when I started, I remember. So, my, you know, just a quick background. I did my bachelor's, and I did my master's, and I was, like, you know, I'm going to academia. I'm not going to the industry, but I was, like, you know, PhD is a big commitment. Let me do an internship before I kind of, like, you know, throw myself into this, like, you know, hell that is doing a PhD. And I ended up really liking it, but I was always more, you know, academically minded. So, I started working, and I started using R because I kind of used it for my thesis, and, like, very quickly people are, like, wait, what are you using? Like, we use SaaS here. We don't use R. Like, this is open source. Like, how do we know it's, like, the right outputs and everything? And I was very lucky because my manager actually, he told me, like, don't listen to them. You do you. You know, you're doing great work. You do you, and we kind of, like, followed down that path.

But as I would say, like, very quickly within, like, the next few years, I think people started seeing, just in general, like, different pockets. Like, you know, I was in a little insurance pocket, but I'm sure the same thing was happening in different pockets. Very quickly people were trying to adopt, starting to adopt these things because it was solving a lot of issues that, you know, like, SaaS was, like, way behind, and, like, you know, by the time you onboard, like, a new tool and you have the license agreements and everything, like, you know, the project would be, like, done and implemented, right? So, that starts to happen, and I think naturally we kind of, like, went through this direction where things started getting accepted, and I think in the meantime what was happening is that there was some leadership changes that were happening, like, enterprise data and analytics team was being created, and we were very lucky, and to this day we're very blessed that, you know, our leadership, they're very forward-thinking. Like, to them, it's a non-issue, and even to this day, like, for our enterprise data and analytics team, we're, and this is probably not super common in the industry, we're really a code-first sort of mentality where, you know, for example, one of the big projects that we have right now is creating our own in-house machine learning end-to-end library, right? So, this is, like, a big project, and we're building this ourselves, and it's using, like, a fully open-source stack.

TD also acquired an AI company. I was one of the first banks, at least in Canada, I was one of the first banks to do so, and then after that, like, other banks started doing that as well a bit. So, I would say we've been very lucky that we kind of organically managed to adopt a lot of these tools, and we had the right leadership in the right places to kind of, you know, allow that. With that said, it's not always without challenge. So, for example, you know, we still have challenges bringing, let's say, third-party data. It's not, sorry, not third-party data, but, like, third-party models, because it's like, well, what do we know about these models, right? Like, what's in these models? So, those challenges still exist, and we have partners in our Enterprise Protect team that actually, you know, they proxy all the

libraries, so we don't have direct access to the libraries. All these libraries get scanned before they get put on our internal repository. So, we do have these layers of control that are kind of put in place, and, you know, again, the other layer of control that we have is any model and stuff that we create, right? We can create models with anything that we want, but if we want that model to go into production, it's not like we can take any package or anything that we want, right? Everything does get vetted by a model validation team. So, that model validation team is like, okay, like, this library that you're using, they look into the documentation, they look at all the methodologies that are used, and they end up vetting this. This can be a very long process for certain models, and, you know, it's grueling, and it's not the funnest thing, but these are the controls that they've essentially had to put in place so that we can do these things in a regulated environment, right?

Absolutely. I appreciate that. You're really painting a good picture for a career in MRO, right?

SAS usage and team structure at TD

There was an anonymous question I want to make sure I didn't forget that was over on Slido but the question was, do people use SaaS at TD? I think it's finally going away. So, I think up until very recently, and maybe still, but I think the goal for the enterprise has been to migrate everything off of Azure. So, I think everything is sort of going to be moving into sort of like this new Databricks platform. So, there might be some small pockets. It's a very big company, so it's hard to know exactly what's going on in every single spot, but very little SaaS, I would say. If it is, it's some legacy stuff that, you know, people just don't want to touch.

Alan, I see you had another question a little bit earlier here. Do you want to jump in? Yeah, thanks. I'm really curious about scaling up to the size of team that you have now, Marin, where you've got people dedicated to this ML-AI space and use cases there.

Was there a point where there was enough support to say, yeah, it makes sense to dedicate people here as opposed to, you know, sort of like catch-as-can models where, you know, you get use cases coming in and you try and solve them, but those folks may be slack for some time because up to a certain point in time, you just don't have those all the time. So, I'm curious what that sort of progression was like and how you knew when you were like, okay, this is enough to focus on, to dedicate FTEs to, as opposed to, you know, sort of filling that in as it happens.

So, just to be sure I answered the right question, you're referring to sort of like AI ML use cases in general or specifically to sort of like some of the stuff that's more like front-end application and things like that? I guess the way I understand your team to be, they're focused on the AI ML use cases. And so, there must be enough of that work. There's enough volume to make sense to have that specialization versus like kind of a big pool of people. And yeah, and one thing that I can maybe like make a precision about my team is that our team actually doesn't even do that much specifically AI ML work in terms of like the model development and stuff like that actually doesn't happen much in our team. So, we work with our partners. So, our team kind of acts as like a central spoke for projects where we kind of coordinate a lot of things. We get things into production. If there's things to build out more engineering focus, we do it. But the actual like AI ML happens in different practices for different lines of businesses.

And the model building itself that there's a huge market for at TD. Like we have like between marketing and between insurance and all the different lines of businesses. The AI ML practice I would say has tons of work. And I think the practice that is really focused on machine learning and model building is around 150 people today. And I would say they're fairly busy. My team kind of focuses more on the end-to-end delivery of some of these like special projects where it's not just like a model, but it requires like a lot of data engineering and it requires a model and it requires let's say like a front end. And in our case, like my team is roughly 11-12 people. We definitely have enough work for us as well to do this. And to be honest, in this space, we really only are catering to insurance for now. Once the other lines of businesses start adopting this model as well, I think this team will need to grow. Because one of the things that we realized is that building a model is one thing, but then making sure that the right decisions are being made based off of these models, that's where that's your last mile delivery. And like how do you want to present this? Tableau is great. And I know a lot of industries use Tableau or Power BI and things like that. But the part that we kind of focus on is like, okay, we don't want to just give you data. We want to also capture data from you. And that's where Posit Connect, where it gives you that flexibility of deploying these proper applications, gives you that.

So I would say, yeah, we do have enough use cases to kind of grow these things. But if we go back, it's been a journey of like a few years. It started a lot smaller, right? Like the team might have been like four people kind of doing like a couple of use cases and then kind of like started organically growing until a point where we started having to create these specialized roles of like a back-end developer, a front-end developer, these things.

Cool. Thanks. It's really, really useful to hear. I really appreciate you fleshing that out. Thanks.

Thank you. Lisa, your puppy is really interested in this conversation too. I think we need an introduction. This is Finny, and he's seven months old. And yeah, so far really loves hearing about he was at Posit Con, you know, from home. So, yep. Loving your conversation.

Working within security frameworks at financial institutions

Marie, I see you asked a question in the chat. I can, if you want to jump in and add some context, feel free. I can read it. I can probably read it. So I think a lot of folks on this call are probably working for financial institutions or have. So I do a lot of consulting for financial institutions. And as, you know, obviously we want to be implementing ML and be, you know, building all of these tools, but I often am at the mercy of whatever security and networks kind of shenanigans that the client might have. So I am subjected to their, you know, security risks. Financial data in particular can be incredibly sensitive. So I wanted to hear about how you might have coordinated or maybe different tooling that you might have used to get around or, you know, work with other teams to build some of the tools that you are using. I'm just kind of getting started on this project and I'm just having a lot of difficulty convincing stakeholders that, you know, this is necessary, data is secure, etc.

Yeah, for sure. So I think it's very important to have like a good understanding of their framework. So I think every larger company will probably have their own sort of, you know, layers of control, right? So there is privacy partners on one side, like they tell you, you know, this is the data I want to use. This is what you're allowed to do with the data. And being able to have those conversations, that's important. And then you have sort of like the frameworks that are sort of like set by whoever is their sort of like security team, right? So we call them enterprise protect. And typically, it's important to understand sort of the guardrails that they've put in place. So in our case, you know, the way that it works is that, okay, well, you have this zone. In this zone, you're allowed to do X, Y, Z. And this is sort of like the guardrails that are put in place. And then if there are existing patterns that, you know, the architecture team has kind of put in place, as long as you stay within those, you're kind of okay. Now, the difficulty happens if you have like a use case that doesn't really fit those patterns. And that's where it becomes challenging. So, you know, I can kind of speak to that, because we had those challenges. Because when we were migrating our Posit Connect platform onto Azure, they were like, well, the Azure environment isn't really made to have data coming into it in this kind of fashion, right? So that, again, was not an easy conversation, because they had to understand exactly what is going on, they had to build new patterns. But once we got to the point that, you know, the patterns were built, all the conversations were had, Enterprise Protect gave their sign off, now you have this pattern that you have that you can reuse and kind of do whatever use case that you need to, right?

So I think the key here is to be able to like abstract your use case into a pattern that you can kind of explain to the partners and say, like, this is what I need to do. And you also have to be kind of like, you know, open at the end of the day that they say, like, okay, well, you know, to make your data secure, you need to do X, Y, Z, and kind of like hear them out as well. So that it's not it's not easy. And especially if I take the consultant point of view, I feel like that's particularly difficult, because you know, you you're kind of like time bound. And some of these things, I'll be honest, we just take a lot of time and a lot of, you know, effort to kind of like, get things going. With that said, the benefit that you have on the consultant side, I think is that, you know, if you're paid by the company, they want to see results. So I would say use your business partners to kind of like, you know, push some of those conversations along with the security partners and say, like, you know, so they understand the value, because a lot of the conversations that we've had, you know, I can see is like, if I go to them with like a use case, and I say, hey, I need to do this, and it requires some new patterns, they're gonna be like, no, we're not going to do this. And essentially, if I look at it from their perspective, like, why am I going to bother creating some potential risk for this thing that you're coming to me, right? So you do need to have that support from the business side that says, like, no, this is important. This is going to enable lots of value for us. And we need your help to tell us like how to do this, but do it within whatever like risk appetite that you have. Once you have that support, it becomes a lot easier to kind of push things along. But you do need to have your business partners there that who are, you know, paying you to say, like, hey, I need you guys to support this, because whatever you're paying me for, I can't do if you know, they don't they don't help me out.

So I think the key here is to be able to like abstract your use case into a pattern that you can kind of explain to the partners and say, like, this is what I need to do.

Thank you for such a thoughtful answer. Gerard, I see you have your hand raised. Want to jump in here? Yeah, sure. So I'm at Prudential and a similar scenario where we like to hear from external partners on how they do things and how they solve issues. And there's there's two things that we use very successfully. One is sandboxes. They're, they're kind of the environment that is in production. But it's completely separate from the corporate network. And there's just nothing you can do to break things. And another thing that we were very good at is making fake data that looks a lot like our data. And then, you know, ask the consultant or the external party to develop a solution that works within that, knowing anyway, and I think I think Marin alluded to this as well, that no matter what we develop, there will be an IT organization that will take that whole thing apart, build it almost from scratch when it has to go into production anyway.

Yeah, that's a that's a really great point. So the sandbox is very key. And this is actually, I think, a concept that, at least for our IT department was a little bit foreign, I would say, like maybe going back like a few years, because to them, it was like, well, you have a dev environment, and then you have like all these like, you know, SIP, PAT, whatever you want to call it. And they have this thing called, you know, SDLC, like the software development life cycle. So what they would do is typically it's like, well, you work with dev data, which is non production, like fake data, and then you do that. But what we eventually convinced them is that, well, we kind of need a dev, but we need a dev that's with like real data, like essentially a sandbox, because working as a data scientist with fake data becomes a little bit difficult, right, when you're actually trying to prove results and things. So, so yeah, so the sandbox idea of like, you know, it, it looks like a production, it breathes like a production, but it's not production. That's something that I think is key for any data science team to be, you know, to be useful, right? So that having being able to have the freedom to do that, yeah, I fully agree. Yeah, that's a great point. So as long as you can kind of create what you need in that environment and show the value and everything, and then whether it's IT that rebuilds it, or you're able to find a different model, that, that kind of can go a long way for sure.

Yeah, good tip for, for, from a data perspective, and it's not an R product, it's, it's Y data. They have a package that is really well in creating fake data by looking at your own data. So it learns from your data and then generates fake data sets. So that allows you to have a little bit more of that production worthy data in a development environment to combat the problem that you were describing. Yeah, that's great actually.

Thank you. Bill, I see you had a question a bit earlier on the model validation team. Do you want to jump in? And Bill, I can, I'll ask the question if you want to add more context, feel free. But it was interesting that you have a model validation team. What kind of members and what skill sets are on that team? Yeah.

Hold on, I'm hearing something in the background. Not sure.

Not sure where, oh, they muted. Okay. Yeah. So the model validation team, very important team, very popular in terms of like anybody that is doing any model, they interact with them. So that team, I would say the, the background of the members, they're typically PhD, very highly skilled in, you know, the academic side of model. Like they kind of understand really like, you know, all like the splitting methods, all like the little detailed methods of, you know, how do you treat your data and what impacts those things can have on your model. That's really their, their skill set. I would say they have a particularly difficult job as well, because first of all, like, you know, everybody's doing models, right? So you have a lot of models coming from different lines of businesses and you can't really look at a model in a vacuum. You need to understand sort of like the business context of the models. And if you have like a centralized team that kind of deals with a lot of different lines of businesses, that means like very quickly learning about those lines of businesses and that particular use case.

So they, they also need that skill set of like, you know, being able to talk to the business, understand what is going on, and then also have a very deep understanding on the technical side of the, of the model. So, so yeah, they're very, very important partner and they're, they're really key on getting, so that's, that's, you know, one layer. And then we have a different layer that is more on the bias and fairness side. So getting the model validated is more of a technical question of like making sure that the model is going to produce the outputs that you expect and everything. But then after that, it's like, well, all these variables and all these things that we used in our models, is it even fair to use those in the context that you're using them, right? So, and that's like a different partner that we, that we work with.

Career advice

Thank you. As we get to the last 10 minutes here, I just want to remind people, if there is a question you asked a bit earlier, anything I missed, feel free to just raise your hand here on Zoom so I can see you as well. But Mehran, a question I love to ask everybody here is, what is one of the most memorable pieces of career advice that you've either received or given throughout your professional journey?

Yeah, that's a good question. So one of the things that, you know, so my, my manager for a long time at TD, so, you know, I worked very closely with him and like, I wouldn't say it's necessarily an advice that he gave me, but it's something that I really learned from him, like working with him and being in meetings with him is, you know, like, if your vision and your goals are set in, you know, in the right way, right? So, and when I say that, you know, pessimistically, you know, you can look at your career and say like, well, you know, I want this job, so I'm going to do whatever it takes to do that. But I think the better way to kind of look at your career is like, well, what do I expect to kind of like achieve, right? And what are sort of like the vision that I have for this company, right? And like, think big and what are the steps that you need to do to kind of achieve that? Like, if you have that kind of a mindset and you do everything, you know, in that kind of direction, you will achieve great things. And not only that, but you'll bound, you'll band together like a group of people, right? And you have a certain motivation because all you're doing isn't like, well, you know, this model is going to bring this benefit, let's just do it and get across the line. It's like, okay, well, you know, this, for example, this call center app, it wasn't that, you know, we wanted to do this and like solve the problem. It was like, we realized like, oh my God, like people are on the hold, you know, it's during COVID for like an hour and like, we need to help them out, right? Like, and we did something similar for the claims team. And it was like people that were like waiting for responses that they weren't getting any. And we realized, okay, well, we need to kind of like do something about this, right? So having that kind of mindset that, you know, at the end of the day, like it's the clients that matter and what's my vision and how do I think my skill sets are going to, you know, help that out. That's sort of like what brought us our team to this point where, you know, that's what we're really doing, right? It's not about the application. It's not about the models or anything. It's really about like, what's the end result of this.

the clients that matter and what's my vision and how do I think my skill sets are going to, you know, help that out. That's sort of like what brought us our team to this point where, you know, that's what we're really doing, right? It's not about the application. It's not about the models or anything. It's really about like, what's the end result of this.

I love that. Thank you so much. So do you give your team an opportunity to like meet, to talk about those big picture things or where do people usually?

Yeah, for sure. So we do, we try to do twice a year, what we call an offsite. So we kind of get together. We don't talk about necessarily sort of like the specifics, but we talk about sort of the division of the team and, you know, sort of some of the past learnings, what went well, what didn't go well, what we want to achieve, like sort of in the coming year. We also kind of combine that with some like activities, just, you know, some team building things. Like last time we were, we kind of got everybody in Montreal and we actually did like an art, like a little art workshop together. So I think that kind of thing is important because a lot of these sort of like discussions about like vision and like about the company and things like that kind of arise from there. And TD does this annual, and I think they're going to make it more frequent, but what they call TD Pulse, which is they sort of get a pulse of everybody. And then we get some of that information and sort of, you know, things about, well, have you in the last year thought about getting a job somewhere else? And we try to talk about those things like very openly and try to, you know, make things better, not just like in terms of our work, but for ourselves, right? Like to be able to produce good work, everybody needs to be happy at the end of the day.

That's great. Thank you. I'm actually headed to Montreal at the end of this month, so I might need to ask you for some suggestions. Yeah. I don't see any questions that I missed yet. I'm going to double check Slido, but I did see there's a lot of conversation, not necessarily questions, but conversations about Databricks in the chat. And I can't read through all the comments right now, but I think there was some people kind of weren't sure really what Databricks was when we were talking about it. Would you be able to give a little bit more insight into like how your team uses that or what it's for?

What is Databricks?

Yeah. So Databricks is essentially like a managed Spark environment. So essentially imagine, you know, you want to have a cluster, like a Spark cluster to work with. Well, Databricks kind of lets you do that. You press a button, you say, I want a cluster of like this many nodes. And then it spins that up using whatever sort of cloud services. So if it's Azure Databricks, it's going to be on Azure, but you can do it on AWS anywhere else. And then you have that cluster sort of set up. So from a data scientist point of view, I would say it's not like, you know, it's nice to be able to have that flexibility so that you can request what you want when you want. With that said, I'm pretty sure no company out there allows their data scientists to spin up clusters willy-nilly because it costs a fortune. But what it does do is that your IT department doesn't need to purchase a bunch of servers and things like that and set it up. So it allows you to kind of like, you know, if you have like a use case, you can say, well, like I have this use case, I need like a GPU cluster for this use case. While, you know, you kind of get your, whoever needs to approve it to say like, okay, yeah, we approve to have this kind of cluster to develop this use case and you have it like, you know, within minutes, right? So that's essentially Databricks and their focus is really on Spark runtime. So that's, I don't know. I don't know if I did a good enough job for Databricks sales, but that's sort of like essentially what it is.

Yeah. And we didn't want to put the job of sales rep on you, but I just saw there was a lot of conversation there in the chat about it. Thank you. That's helpful.

Spreading the word internally

I wanted to maybe end on something you had said to me when we were having our pre-call is you explained some like examples your team or use cases your team had kind of built out with the business and people saw it and were like, why aren't we doing this already? And like, how is insurance doing this? And why aren't we doing this? And I would just love to hear a little bit about like that process internally. Like how do you celebrate some of the work that is being done and when people want to do something similar, how do they go about doing it and learning how to do that?

Yeah, for sure. So one of the things that you know, we realized is when we do things for sort of like our business partners and at the end of the day, they're mainly the users here. They will go and sell the thing for us. Like we have the executives on our claim side that goes and says like, this is the next best thing since sliced bread and they go and talk to other executives and that's where a lot of these things come from. So we have a lot of support on the business side and that's kind of comes naturally because they like what we do and they like what we do because we do pretty much exactly what they want targeted to them, right? So that kind of gets the word out like very quickly from like a top down kind of way. And from a bottom up kind of way, we do have these like very similar to this event actually, where we come and we do short 15 minute presentations of a use case to the very larger group of like enterprise data and analytics teams. So like I said, like this is upwards of like 4,000 people and that starts a lot of these conversations that are like, well, how do we do that? And for example, you know, initially TD Insurance was using Posit Connect quite a bit. That's where it kind of like started off. But then, you know, our wealth partners saw this and they're like, hey, like we have a bunch of use cases that would be great for that. We talked together and we're like, oh yeah, you know, and now, you know, we have our personal banking side. So things kind of start growing organically. But yeah, these sort of events are really useful to connect us, especially when we become like a very big community of people.

Conference recommendations

Absolutely. I see Gerard asked one last question here. Any advice for fintech conferences or what kinds of conferences are you interested in?

Yeah, so I get, I don't know how many emails I get that ask me to go eat steak in New York City and sit next to like a CFO or a CIO or a new ML guru or whatever. They're all really, really pushy. Any conferences that you feel are very much in the fintech and the insurance banking industry that are for practitioners and not for, I guess, posers?

Yeah, that's a good question. Because, you know, there's the big insurtech in Vegas, I think that that one is, I think I haven't been to that one, but I know a lot of the partners on the business side that I work with. That's basically the only one that they end up going to, because that's where, you know, you get most of the vendors that are interesting and every other one kind of becomes very samey. But that is very more, that's like very like sales focused, I would say, like, it's a higher level. On the more practitioner focused, I'll be honest, I haven't seen many like great sort of conferences. I think, you know, the Posit Conf one being actually like a pretty good one. And they're kind of like far and few in between. I know Anaconda does something very similar. So that might be, you know, something. But if you know of any, please let me know because I've kind of struggled because some of these more sales focused ones, we do have people that go to them. But I always want to send my team to some of these more interesting ones that are practitioner focused. And they're kind of like few and far in between. Posit Conf being one of the rare ones. But yeah, there isn't that many.

I guess I would love to ask you all, then what would you like to see? That would help me. Help me in planning too. Not that we're planning any steak dinners, but I don't know. A lot of these conferences seem to be focused on like one or the other where either you have people that are sort of, you know, selling, you know, these platforms and things like that, or you have, you know, smaller meetups that are like very technical focus in like a very specific kind of area. I think having a conference where you kind of see, you know, some end to end projects where, you know, there's the development portion of it, but there's also like some talks about like how this thing even got deployed, right? Understanding what that flow kind of looks like for people. I think that that's one of the things that would be very interesting to see.

Cool. Well, thank you all so much for joining us today. Thank you so much, Mehran, for sharing your experience with us all. Kicking us off in 2024. Have a great rest of the day, everybody.

Thank you.